Methods and Systems for Rendering Virtual Reality Content Based on Two-Dimensional (2D) Captured Imagery of a Three-Dimensional (3D) Scene

ABSTRACT

An exemplary method includes a virtual reality content rendering system receiving two-dimensional (“2D”) color data and depth data captured by a plurality of capture devices disposed at different vantage points in relation to a three-dimensional (“3D”) scene, receiving metadata, generating, for each vantage point associated with each respective capture device included in the plurality of capture devices, and based on the metadata and the depth data, a partial 3D mesh projected into a virtual 3D space to produce a partial representation of the 3D scene in the virtual 3D space, and generating, based on the partial 3D meshes projected into the virtual 3D space, and from an arbitrary viewpoint within the virtual 3D space, an image view of the virtual 3D space. The generating of the image view may include blending color samples of overlapping sections off the partial 3D meshes projected into the virtual 3D space.

RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 15/610,586, filed May 31, 2017, and entitled“Methods and Systems for Rendering Virtual Reality Content Based onTwo-Dimensional (“2D”) Captured Imagery of a Three-Dimensional (“3D”)Scene,” which is hereby incorporated by reference in its entirety.

BACKGROUND INFORMATION

Advances in computing and networking technology have made new forms ofmedia content possible. For example, virtual reality media content isavailable that immerses viewers (or “users”) into interactive virtualreality worlds that the users may experience by directing theirattention to any of a variety of things being presented in the virtualreality world at the same time. At any time during the presentation ofthe virtual reality media content, a user experiencing the virtualreality media content may look around and/or move within the virtualreality world in any direction with respect to both a horizontaldimension (e.g., forward, backward, left, right, etc.) and a verticaldimension (e.g., up, down, etc.), giving the user a sense that he or sheis actually present in and experiencing the virtual reality world from aparticular viewpoint within the virtual reality world.

In some examples, a virtual reality media provider may provide virtualreality content that includes a virtual reality world by transmitting,by way of a network, data representative of the virtual reality world toa client computing device being used by a user to experience the virtualreality world.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary virtual reality content rendering systemaccording to principles described herein.

FIG. 2 illustrates an exemplary configuration in which a renderingfacility of the system of FIG. 1 receives and processes data associatedwith a 3D scene to render virtual reality content representative of the3D scene according to principles described herein.

FIG. 3 illustrates an exemplary configuration of capture devices withrespect to an object in a 3D scene according to principles describedherein.

FIG. 4 illustrates exemplary two-dimensional (“2D”) input meshesaccording to principles described herein.

FIGS. 5-7 illustrate exemplary projections of 2D input meshes to depthswithin a virtual 3D space according to principles described herein.

FIGS. 8-10 illustrate exemplary operations for identifying and removingextraneous primitives from a partial 3D mesh according to principlesdescribed herein.

FIGS. 11-12 illustrate cross-sectional views of a plurality of partial3D meshes projected into virtual 3D space according to principlesdescribed herein.

FIG. 13 illustrates a cross-sectional view of partial 3D meshesprojected into a virtual 3D space and that may be used to generate animage view of the virtual 3D space from an arbitrary viewpoint accordingto principles described herein.

FIG. 14 illustrates an exemplary configuration of frame buffersaccording to principles described herein.

FIG. 15 illustrates an exemplary depth offset of a partialrepresentation of an object to be rendered from an arbitrary viewpointwithin a virtual 3D space according to principles described herein.

FIG. 16 illustrates an exemplary implementation of the virtual realitycontent rendering system of FIG. 1 according to principles describedherein.

FIG. 17 illustrates an exemplary transport stream that may be receivedby the virtual reality content rendering system of FIG. 1 according toprinciples described herein.

FIG. 18 illustrates exemplary 2D color and depth data that may bereceived by the virtual reality content rendering system of FIG. 1 froma particular capture device during a particular temporal sequenceaccording to principles described herein.

FIG. 19 illustrates an exemplary implementation for metadata accordingto principles described herein.

FIG. 20 illustrates an exemplary method for rendering virtual realitycontent based on 2D captured imagery of a 3D scene according toprinciples described herein.

FIG. 21 illustrates another exemplary method for rendering virtualreality content based on 2D captured imagery of a 3D scene according toprinciples described herein.

FIG. 22 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Methods and systems for rendering virtual reality content based on 2Dcaptured imagery of a 3D scene are described herein. In certainexemplary methods and systems, a virtual reality content renderingsystem (“rendering system”) may receive data representative of 2Dimagery of a 3D scene (e.g., a real-world scene) as captured by aplurality of capture devices (e.g., video cameras, 3D depth scanninghardware, etc.) physically disposed at different vantage points inrelation to the 3D scene (e.g., at different positions having differentcapture angles in relation to the 3D scene). The captured 2D imagery,which may be referred to as surface data, may include 2D color data anddepth data for surfaces of one or more objects in the 3D scene, ascaptured by each capture device included in the plurality of capturedevices.

The 2D color data and depth data captured by each of the capture devicesmay represent a distinct unmeshed view of the 3D scene from a particularvantage point relative to the 3D scene. As used herein, a “distinctunmeshed view” corresponds to a view of a 3D scene from a particularcapture device in which the surface data for the 3D scene (e.g., the 2Dcolor data and depth data) captured by the particular capture device isnot represented as a 3D mesh of geometric primitives and/or is notcombined (i.e., not meshed) together with the surface data captured byanother capture device to form a 3D model or 3D representation of the 3Dscene.

The rendering system may also receive metadata such as projectionmetadata for the 2D color data and depth data. The metadata may includeinformation associated with the 3D scene, such as information about theplurality of capture devices, that is usable by the rendering system toproject the 2D color data and depth data into a virtual 3D space toproduce virtual representations of the 3D scene in the virtual 3D spacesuch that the projected data may be used by the rendering system torender a view of the virtual 3D space (i.e., a view of the virtualrepresentations of the 3D scene in the virtual 3D space) from anarbitrary viewpoint within the virtual 3D space.

Based on the received data (e.g., the metadata and the depth data), therendering system may generate, for each vantage point associated witheach respective capture device, a partial 3D mesh projected into avirtual 3D space to produce a partial virtual representation of the 3Dscene in the virtual 3D space. In certain examples, the generating ofeach partial 3D mesh may include the rendering system projecting, foreach vantage point associated with each respective capture device, a 2Dinput mesh along perspective rays to depths within the virtual 3D spacebased on the depth data to produce 3D coordinates of vertices ofprimitives of the partial 3D mesh in the virtual 3D space. In certainexamples, the generating of each partial 3D mesh may also include therendering system identifying, for each partial 3D mesh, one or more ofthe primitives of the partial 3D mesh as extraneous primitives andremoving the identified extraneous primitives from each partial 3D mesh.

Based on the partial 3D meshes projected into the virtual 3D space forthe vantage points associated with the capture devices, the renderingsystem may generate, from an arbitrary viewpoint within the virtual 3Dspace, an image view of the virtual 3D space. In certain examples, thegenerating of the image view of the virtual 3D space includes therendering system accumulating the partial 3D meshes projected into thevirtual 3D space and blending, based on the 2D color data, color samplesfor the partial 3D meshes to form the image view of the virtual 3Dspace. In certain examples, the accumulating of the partial 3D meshesprojected into the virtual 3D space may include accumulating the partial3D meshes in a frame buffer of a graphics processing unit (“GPU”) andadditively blending the color samples for the partial 3D meshes in theframe buffer of the GPU. In certain examples, the blending of the colorsamples may include the rendering system determining blend weights forthe primitives of the partial 3D meshes, weighting the color samples forthe partial 3D meshes based on the determined blend weights for theprimitives of the partial 3D meshes, and determining weighted averagesof the weighted color samples for superimposed primitives included inoverlapping sections of the partial 3D meshes. In certain examples, therendering system may determine an accumulation region and accumulate andblend only primitives or fragments that are located within theaccumulation region.

Illustrative examples of the above-stated principles and operations, aswell as additional and/or alternative principles and operations, aredescribed herein. For sake of clarity and explanation, certainillustrative examples are described and illustrated herein usingsimplified data sets (e.g., data sets representing a simple 3D scene,object, 2D input mesh, partial 3D mesh, etc.). However, it will beunderstood that the principles and operations described herein may beapplied to larger and/or more complex data sets.

Methods and systems described herein provide one or more advantagesand/or benefits over conventional virtual reality content renderingtechnologies. For example, methods and systems described herein maysignificantly reduce computing resources (e.g., memory and/or processingresources) required to render virtual reality content. To illustrate, byusing 2D color data, depth data, and metadata to generate, accumulate,and blend partial 3D meshes to form an image view of a virtual 3D spaceas described herein, a rendering system can render virtual realitycontent (e.g., the image view of the virtual 3D space from an arbitraryand dynamically selectable viewpoint in the virtual 3D space) withouthaving to receive or generate a complete 3D model of a 3D scene, such asa single, cohesive geometric mesh of the 3D scene. This results in asavings of transmission bandwidth, processing requirements, and memory.

Additionally or alternatively, by using 2D color data, depth data, andmetadata to generate, accumulate, and blend partial 3D meshes to form animage view of a virtual 3D space as described herein, methods andsystems described herein may use existing video processing technologies(e.g., video rendering technologies) and existing graphics hardware(e.g., existing graphics cards and pipelines) and/or software to rendervirtual reality content, which may support scalability of virtualreality content services to a wider range of end-user computing devicesand/or expand the accessibility of virtual reality content to users ofvarious end-user computing devices.

Additionally or alternatively, by using 2D color data, depth data, andmetadata to generate, accumulate, and blend partial 3D meshes to form animage view of a virtual 3D space as described herein, methods andsystems described herein may render virtual reality content faster thanthe virtual reality content would be rendered using conventionalrendering technologies. This may support applications in which virtualreality content is streamed and rendered in real time or near real time,such as for virtual reality content representing an event as the eventoccurs in real time.

Additionally or alternatively, by using 2D color data, depth data, andmetadata to generate, accumulate, and blend partial 3D meshes to form animage view of a virtual 3D space as described herein, methods andsystems described herein may facilitate significant reduction in theamount of bandwidth required to provide virtual reality content to amedia player device by way of a network. For example, instead ofgenerating a 3D model of a 3D scene prior to transmission, 2D datarepresentative of distinct unmeshed views of a 3D scene may be generatedand transmitted to a media player device by way of a network. Thissignificantly reduces the amount of data that is required to beprocessed server-side and transmitted over the network, which may enablestreaming of virtual reality content at increased speeds with reducedprocessing and/or bandwidth requirements, which may improve datathroughput and latency.

The systems and methods described herein may provide additional oralternative benefits as may serve a particular implementation. Variousembodiments will now be described in more detail with reference to thefigures. The disclosed methods and systems may provide one or more ofthe benefits mentioned above and/or various additional and/oralternative benefits that will be made apparent herein.

FIG. 1 illustrates an exemplary virtual reality content rendering system100 (“system 100”) that may process 2D color data, depth data, andmetadata to generate, accumulate, and blend partial 3D meshes to form animage view of a virtual 3D space in accordance with methods and systemsdescribed herein. As shown, system 100 may include, without limitation,a rendering facility 102 and a storage facility 104 selectively andcommunicatively coupled to one another. In certain alternative examples,storage facility 104 may be excluded from and communicatively coupled tosystem 100. It will be recognized that although facilities 102 and 104are shown to be separate facilities in FIG. 1, facilities 102 and 104may be combined into fewer facilities, such as into a single facility,or divided into more facilities as may serve a particularimplementation. Each of facilities 102 and 104 may include or be housedin a device (e.g., a media player device having a single chassis) andlocated at a single location (e.g., a user premises such as a residence,a business, etc. or another location associated with a user of a mediaplayer device) or distributed between multiple devices and/or multiplelocations as may serve a particular implementation. Each of facilities102 and 104 will now be described in more detail.

Storage facility 104 may store 2D color and depth data 106, metadata108, virtual reality content data 110, and/or any other data received,generated, managed, maintained, used, and/or transmitted by renderingfacility 102. 2D color and depth data 106 may include 2D color data anddepth data that may be received by rendering facility 102 and that isrepresentative of one or more objects in a 3D scene, such as theappearance and location of the objects in the 3D scene.

In certain examples, the 3D scene may be a real-world scene, such as a3D scene of a real-world event that may be occurring live (e.g., in realtime). In such examples, the 3D scene may include one or more real-worldobjects associated with the real-world event, and 2D color and depthdata 106 may be acquired in real time as the real-world event occurs. Inother examples, the 3D scene may be a virtual scene that includes one ormore virtual objects. In other examples, the 3D scene may be amerged-reality scene, and the 3D scene may include at least one virtualobject and at least one real-world object. The 3D scene may beconsidered a 3D scene because each object and/or surface in the 3Dscene, whether real or virtual, has a detectable position in 3D spacewith respect to the 3D scene and other objects in the 3D scene.

As used herein, an “object” may include anything that is visible (i.e.,non-transparent) from a particular vantage point in a 3D scene, whetherliving or inanimate. For example, if the 3D scene corresponds to areal-world event such as a basketball game, objects may include thebasketball being used for the game, the basketball court, the basketballstandards (i.e., the backboards, rims, nets, etc.), the players andreferees participating in the game, and/or any other object present atand/or associated with the basketball game.

As used herein, “2D color data” may broadly include any datarepresentative of how a 3D scene appears from at least one vantage pointof at least one device capturing the 2D color data. The 2D color datamay include a color (e.g., red, green, blue, etc.) representation of the3D scene, a black and white representation of the 3D scene, a greyscalerepresentation of the 3D scene, and/or any other suitable representationof the appearance of the 3D scene. In certain examples, the 2D colordata may include a captured sequence of images (e.g., high-resolutionstill images) representative of the appearance of the 3D scene (e.g., anobject at a real-world event) from a vantage point over a particulartime period. For instance, the 2D color data may include 2D color videodata of the 3D scene in which a frame of video represents color valuesat pixel coordinates of the frame. However, the 2D color data is notlimited to any particular format, file type, frame rate, resolution,quality level, or other characteristic that may be associated withvarious definitions and/or standards defining video in the art.

As used herein, “depth data” may broadly include any data representativeof spatial positions of one or more objects (e.g., one or more objectsat a real-world event) within the 3D scene. The depth data may beformatted in any suitable manner. In certain examples, as will bedescribed below, the depth data may be formatted as a series ofsequential 2D depth data captures (e.g., separate instances of depthdata captured at particular times) from a particular vantage point. Forinstance, the depth data may include 2D depth video data of the 3D scenefrom a particular vantage point and in which a frame of video representsdepth values at pixel coordinates of the frame. However, the depth datais not limited to any particular format, file type, frame rate,resolution, quality level, or other characteristic that may beassociated with various definitions and/or standards defining video inthe art. Preferably, the depth data has high precision and accuracy.However, the depth data may have generally lower resolution (e.g., lowerx and y coordinate resolution) than the 2D color data and still beacceptable.

The depth data may be synchronized with the 2D color data. For example,a depth data frame in the depth data and a color data frame in the colordata may correspond with a common instance in time (e.g., a samereference signal, timestamp, capture time, etc.). Such a correspondingdepth data frame and color data frame may form a pair of synchronizedframes that, together with synchronized pairs of other depth and colordata frames captured from other vantage points, and with correspondingmetadata, may be processed by rendering facility 102 in any suitablemanner, such as described herein, to generate, accumulate, and blendpartial 3D meshes to form an image view of a virtual 3D space.

The 2D color data and the depth data may be captured by a plurality ofcapture devices disposed at different vantage points in relation to a 3Dscene. The capture devices may include, for example, 2D video cameras,3D depth scanners, combination capture devices (e.g., devices configuredto capture both 2D video and associated depth data), and so forth. Eachcapture device included in the plurality of capture devices may capture2D color data and depth data for the 3D scene from a particular vantagepoint in relation to the 3D scene. As used herein, a “vantage point”refers to a position and capture angle at which surface data of a 3Dscene is captured by each capture device included in the plurality ofcapture devices. In certain examples, the vantage point of a particularcapture device may be fixed (i.e., the position of the particularcapture device and the capture angle of the particular capture device donot change over time). Alternatively, one or more of the position andthe capture angle of a particular capture device may change over time.For example, the position and/or the capture angle of a particularcapture device may change over time as an apparatus to which theparticular capture device is fixed moves in relation to the 3D scene. Avantage point may be described by any other suitable information as maysuit a particular implementation. Exemplary vantage points of capturedevices are described herein.

The plurality of capture devices may capture 2D color data and depthdata in any suitable manner and using any suitable devices as may servea particular implementation. In certain examples, the capture devicesmay consist of video cameras or other types of image capture devicesthat may capture 2D color data of objects in a 3D scene from multiplevantage points and from which depth data for the surfaces of the objectsmay be captured (e.g., derived) by using one or more depth capturetechniques (e.g., triangulation-based depth capture techniques,stereoscopic depth capture techniques, etc.).

In other examples, the capture devices may include video cameras orother types of image capture devices configured to capture the 2D colordata, as well as separate depth capture devices configured to capturethe depths of the surfaces of the objects using one or more of the depthcapture techniques (e.g., time-of-flight-based depth capture techniques,infrared-based depth capture techniques, etc.). For example, eachcapture device may include a first component (e.g., a video cameradevice) configured to capture 2D video of objects at which the firstcomponent is directed, and a second component (e.g., a depth cameradevice, a 3D imaging or 3D scanning device, etc.) configured to capturedepth data of objects at which the second component is directed. In thisexample, the first component and the second component may be separate ordiscrete devices, but may be communicatively coupled and configured towork in conjunction with one another to synchronously (e.g.,simultaneously) capture both the 2D color data and the depth data.

In other examples, each capture device may comprise a combinationvideo-depth capture device (e.g., a specially-designed video camera)that is configured to capture both the 2D color data and the depth data.The combination video-depth capture device may be a commerciallyavailable or specially-designed video camera capable of not onlycapturing video data of objects in a 3D scene but also detectingcorresponding depths of the objects using one or more suitable depthcapture techniques.

In some examples, the capture devices may have finite capture angles(e.g., 90 degrees, 120 degrees, etc.) and may be positioned and directedto capture data from respective areas of a 3D scene. For example, a ringconfiguration of capture devices may be positioned to surround a 3Dscene or one or more portions of the 3D scene (e.g., a basketball court,turns on a racetrack) and be pointed inwardly to capture datarepresentative of objects in the 3D scene. In the same or otherexamples, at least one particular capture device may have a 360-degreecapture angle and may be positioned to capture data from objectssurrounding the particular capture device. For example, at least one ofcapture devices may be a 360-degree camera configured to capture and/orgenerate a 360-degree video image of the 3D scene around a center pointcorresponding to the 360-degree camera. While certain exemplaryconfigurations of capture devices relative to a 3D scene are describedherein, the examples are illustrative only. Any suitable configurationof capture devices relative to a 3D scene may be used in otherimplementations.

The plurality of capture devices may be communicatively coupled to oneanother (e.g., networked together) and/or communicatively coupled toanother device or system (e.g., system 100) in any suitable manner, suchas described herein. This may allow the capture devices to maintainsynchronicity in time, position, angle, etc. For example, the capturedevices may send and receive timing signals to ensure that each capturedevice captures corresponding data at a common time (e.g., within athreshold range of time) and that the data captured by different capturedevices may be timestamped with a universal time shared by all of thecapture devices. In certain examples, a controller may direct thecapture devices as to when to capture and/or output data for a 3D scene.In other examples, the capture devices may control when data for a 3Dscene is captured and/or output, and any suitable facility may sortand/or selectively use the data received from the capture devices. Theseexamples of how capture devices may function synchronously are exemplaryonly. Any suitable device synchronization and/or communicationtechnologies may be used in other implementations to facilitateoperations of capture devices to synchronously capture data for a 3Dscene. In certain alternative examples, one or more of the plurality ofcapture devices may function asynchronously and the data captured forthe 3D scene may be processed and synchronized in any suitable mannerafter capture.

Metadata 108 may include any information that may be used by renderingfacility 102, together with 2D color data and depth data for a 3D scene,to generate virtual representations of the 3D scene in a virtual 3Dspace and to form an image view of the virtual 3D space from anarbitrary viewpoint within the virtual 3D space. For example, metadata108 may include projection metadata that may include any informationthat may be used by rendering facility 102, together with 2D color dataand depth data for a 3D scene, to project 2D color data and depth datafor the 3D scene into a common virtual 3D space to generate virtualrepresentations of the 3D scene in the virtual 3D space. Metadata 108may include, but is not limited to, information indicating temporal andspatial information associated with 2D color data for the 3D scene, suchas when the 2D color data was captured, a vantage point from which the2D color data was captured, which capture device captured the 2D colordata, etc. In addition, metadata 108 may include information indicatingtemporal and spatial information associated with depth data for the 3Dscene, such as when the depth data was captured, where the depth datawas captured, a vantage point from which the depth data was captured,which capture device captured the depth data, etc. Metadata 108 may alsoinclude field of view information (e.g., focal length, optical center,view angles, tangents of angles, etc.), depth mapping information,position information, orientation information, view angle information,translation information, transformation matrix information, changes inany of such information for each capture device, and/or informationabout any other intrinsic and/or extrinsic properties of the capturedevices. In some examples, metadata 108 may include information aboutcoordinate systems (e.g., local coordinate systems associated with thecapture devices and/or scene), projection parameters, projectioncalculations, lens distortion parameters, and/or any other informationuseful for processing 2D color and depth data in any of the waysdescribed herein. In certain examples, metadata 108 may also includepositional information for audio samples captured with respect to the 3Dscene. Metadata 108 may be represented in any suitable format as mayserve a particular implementation, such as Java Script Object Notation(“JSON”), Extensible Markup Language (“XML”), or the like. Exemplaryformats for metadata 108 are described herein.

Immersive virtual reality content data 110 may include any datarepresentative of virtual reality content generated by renderingfacility 102 based on 2D color data, depth data, and metadata associatedwith a 3D scene. For example, virtual reality content data 110 mayinclude data representing generated partial 3D meshes projected into avirtual 3D space, data representing generated image views of the virtual3D space from arbitrary viewpoints within the virtual 3D space, and/orany other suitable data that may be generated by rendering facility 102to provide an image view of a virtual representation of a 3D scene in avirtual 3D space from an arbitrary viewpoint within the virtual 3Dspace. Examples of such data are described herein. In certain examples,virtual reality content data 110 may also include audio data associatedwith a generated virtual representation of a 3D scene, and/or any othersuitable data associated with a generated virtual representation the 3Dscene. Storage facility 104 may further include any other data as may beused by rendering facility 102 to form an image view of a virtualrepresentation of a 3D scene in a virtual 3D space from an arbitraryviewpoint within the virtual 3D space as may serve a particularimplementation.

Rendering facility 102 may include any hardware and/or software (e.g.,computing systems, software programs, graphics processing hardware,etc.) used for generating, accumulating, and blending partial 3D meshesbased on 2D color data, depth data, and metadata to form an image viewof a virtual 3D space. In certain examples, rendering facility 102 mayinclude a graphics card having dedicated video decoding hardware (e.g.,one or more dedicated video decoders) and a programmable GPU.

Rendering facility 102 may receive 2D color data and depth data capturedby a plurality of capture devices disposed at different vantage pointsin relation to a 3D scene. Rendering facility 102 may also receivemetadata (e.g., metadata 108) for the 2D color data and the depth data.Rendering facility 102 may receive 2D color data, depth data, andmetadata in any suitable manner and from any suitable source. Toillustrate, FIG. 2 shows an exemplary configuration 200 in whichrendering facility 102 receives data from a plurality of capture devices202 (e.g., capture devices 202-1 through 202-n) that are physicallydisposed at different vantage points (e.g., vantage points 204-1 through204-n) in relation to a 3D scene 206 that includes an object 208.

The plurality of capture devices 202 may be disposed (i.e., located,installed, etc.) at fixed positions with respect to 3D scene 206 in anyway that may serve a particular implementation. For example, capturedevices 202 may be located at fixed positions surrounding a real-worldevent or one or more portions of the real-world event (e.g., surroundingor partially surrounding a field of play of a sporting event such as abasketball court at a basketball game).

In the example shown in FIG. 2, each of vantage points 204 provides adifferent limited view of 3D scene 206 and of object 208 in 3D scene206. For example, vantage point 204-2 of capture device 202-2 mayprovide a view from above object 208. From vantage point 204-2, capturedevice 202-2 may not be capable of capturing 2D color data and depthdata for a bottom surface of object 208. However, capture device 202-1may be positioned below object 208 and able to capture 2D color data anddepth data for the bottom surface of object 208 from vantage point204-1. Accordingly, capture devices 202 may be configured to, incombination, capture 2D color data and depth data for all of thesurfaces of object 208. Capture devices 202 may be communicativelycoupled to each other and to rendering facility 102 in any suitablemanner, such as described herein.

As shown in FIG. 2, rendering facility 102 may receive 2D color data210, depth data 212, and metadata 214. Rendering facility 102 mayreceive 2D color data 210, depth data 212, and metadata 214 directly orindirectly from capture devices 202 using any suitable datacommunication channels and/or technologies. In certain examples,rendering facility 102 may receive 2D color data 210, depth data 212,and metadata 214 from capture devices 202 through one or more suitabledata communication networks. For example, rendering facility 102 mayreceive at least some of 2D color data 210, depth data 212, and metadata214 by way of a content delivery network (“CDN”) such a CDN capable ofcarrying streaming video and/or other data (e.g., a CDN operated byAmazon Web Services). Exemplary data communication networks throughwhich rendering facility 102 may receive 2D color data 210, depth data212, and metadata 214 are described herein.

2D color data 210, depth data 212, and metadata 214 received byrendering facility 102 may be represented using any suitable dataformats and/or data communication protocols, formats, and technologies.Examples of such data formats and communication streams are describedherein.

Based on received data for a 3D scene (e.g., metadata and depth data fora 3D scene), rendering facility 102 may generate, for each vantage pointassociated with each respective capture device associated with the 3Dscene, a partial 3D mesh projected into a virtual 3D space to produce apartial representation of the 3D scene in the virtual 3D space. Withrespect to FIG. 2, for example, based on the received metadata 214 andthe received depth data 212, rendering facility 102 may generate, foreach of the vantage points 204 respectively associated with each of thecapture devices 202, a partial 3D mesh projected into a virtual 3D spaceto produce a partial representation of 3D scene 206 (and thus object208) in the virtual 3D space.

As used herein, a “virtual 3D space” may represent a 3D space of avirtual 3D world and may be represented based on a set of virtual 3Dworld coordinates. The virtual 3D space may be defined to be anysuitable 3D space that can accommodate partial 3D meshes projected intothe 3D space to produce partial virtual representations of a captured 3Dscene in the 3D space. In certain examples, rendering facility 102 maydefine the virtual 3D space based at least in part on metadata such asinformation about a depth mapping range, a field of view, and atransformation matrix for each vantage point of each capture device.

A partial 3D mesh projected into a virtual 3D space may include a meshof geometric primitives (e.g., triangles, points, lines, polygons,complex geometric shapes, etc.) interconnecting and/or formed by a setof vertices projected to 3D coordinates within the virtual 3D space. Assuch, a partial 3D mesh projected into the virtual 3D space based ondepth data for a 3D scene may produce a partial virtual representationof the 3D scene (e.g., 3D scene 206), and thus of one or more objectswithin the 3D scene (e.g., object 208 within 3D scene 206). The partialvirtual representation of the 3D scene may be referred to as a partialvirtual reconstruction of the 3D scene and may virtually representsurfaces of one or more objects in the 3D scene. A “partial 3D mesh,” asused herein, is a partial 3D mesh, and not a single, cohesive 3D meshfor an entire 3D scene, in that the partial 3D mesh represents only apartial view of a 3D scene, a view from a single vantage pointassociated with a single respective capture device. As described herein,rendering facility 102 may generate and use a partial 3D meshindividually or a plurality of partial 3D meshes to render virtualreality content, without having to construct or use a complete 3D modelof a 3D scene, such as a single, cohesive 3D mesh of a 3D scene.Examples of partial 3D meshes that may be generated by renderingfacility 102 are described herein.

In certain examples, the partial 3D mesh projected into the virtual 3Dspace may produce a partial geometric representation of a captured 3Dscene in the virtual 3D space without color samples from the 2D colordata for the captured 3D scene associated with the partial 3D mesh. Inother examples, the partial 3D mesh projected into the virtual 3D spacemay produce a partial geometric representation of a captured 3D scene inthe virtual 3D space with color samples from the 2D color data for thecaptured 3D scene associated with the partial 3D mesh. Thus, in certainexamples, the generating of the partial 3D mesh projected into thevirtual 3D space may be further based on the received 2D color data 210to associate color samples from the 2D color data 210 with verticesand/or primitives of the partial 3D mesh projected into the virtual 3Dspace as part of the generating of the partial 3D mesh. In otherexamples, rendering facility 102 may perform one or more operationsseparate from the generating of the partial 3D mesh (after the partial3D mesh is generated) to associate color samples from the 2D color data210 with vertices and/or primitives of the partial 3D mesh projectedinto the virtual 3D space.

Rendering facility 102 may generate a partial 3D mesh for each vantagepoint of a 3D scene in any suitable manner. For example, renderingfacility 102 may instantiate in memory and project a 2D input mesh alongperspective rays to depths within a virtual 3D space based on the depthdata received by rendering facility 102. A 2D input mesh may include aset of vertices arranged in a grid of normalized coordinates in a 2Dplane to form a set of geometric primitives interconnecting thevertices. For example, the 2D input mesh may include a 2D grid ofvertices and geometric primitives (e.g., triangles, points, lines,polygons, complex geometric shapes, etc.) formed by interconnecting thevertices with edges and covering an arbitrary, predetermined range(e.g., a signed, normalized range, [−1, 1] that may simplify one or morecomputations). Exemplary 2D input meshes are described herein.

Rendering facility 102 may project a 2D input mesh along perspectiverays, from a vantage point, to depths within a virtual 3D space based ondepth data received by rendering facility 102 and associated with thevantage point. For example, rendering facility 102 may project verticesof the 2D input mesh to depths within the virtual 3D space to determine3D coordinates of the vertices within the virtual 3D space. Theprojection of the vertices of the 2D input mesh to the 3D coordinatesmay determine shapes, orientations, depths, etc. of the primitives ofthe 2D input mesh and may convert the 2D input mesh to a partial 3D meshthat is projected into the virtual 3D space such that the primitives ofthe virtual 3D mesh are positioned within the virtual 3D space tovirtually represent surfaces of one or more objects included in thecaptured 3D scene. In certain examples, the projection of the verticesof a 2D input mesh within a virtual 3D space includes transforming 2Dcaptured imagery (e.g., 2D color data and/or depth data received byrendering facility 102) into a 3D world coordinate system associatedwith the virtual 3D space (e.g., using a transformation matrix includedin metadata received by rendering facility 102).

Examples of rendering facility 102 generating partial 3D meshes will nowbe described in reference to FIGS. 3-10. FIG. 3 shows capture devices202-1 through 202-3 disposed at different respective vantage points204-1 through 204-3 from which capture devices 202-1 through 202-3 maycapture depth data and 2D color data associated with certain surfaces302 (e.g., surfaces 302-1 and 302-2) of object 208 within 3D scene 206.For simplicity of explanation, three capture devices 202 are illustratedin FIG. 3, and FIG. 3 is a cross-sectional view of object 208 within 3Dscene 206 in relation to three capture devices 202-1 through 202-3having three respective vantage points 204-1 through 204-3 relative toobject 208. Thus, the relationships between cube 208, capture devices202, and vantage points 204 are illustrated in two dimensions (e.g.,within an x-z plane of 3D scene 206 represented by a x-y-z coordinatesystem) in FIG. 3. However, it is understood that any number of capturedevices 202 may be provided in relation to object 208 and/or 3D scene206 as may suit a particular implementation, and that principlesdescribed in relation to the 2D cross-sectional view apply to 3Drelationships between object 208 and capture devices 202 having variousvantage points 204.

Rendering facility 102 may receive 2D color data and depth data capturedby capture devices 202-1 through 202-3, as well as metadata for the 2Dcolor data and depth data, and use the received data to generate apartial 3D mesh for each of the vantage points 204-1 through 204-3 ofeach of the capture devices 202-1 through 202-3. To generate the partial3D meshes, rendering facility 102 may generate a 2D input mesh for eachof the vantage points 204-1 through 204-3, such as by instantiating each2D input mesh in memory. FIG. 4 shows exemplary 2D input meshes 402(e.g., 2D input meshes 402-1 through 402-3) that may be generated byrendering facility 102. In the example shown in FIG. 4, 2D input mesh402-1 is associated with vantage point 204-1, 2D input mesh 402-2 isassociated with vantage point 204-2, and 2D input mesh 404-3 isassociated with vantage point 204-3. In the illustrated example, each 2Dinput mesh is a grid of triangles arranged in a 2D plane, each triangle(e.g., triangle 404) represents a primitive (i.e. a face having ageometric shape), and each point of the triangles represents a vertex(e.g., vertex 406) located at a coordinate (e.g., a normalizedcoordinate) in the 2D plane. In the example shown in FIG. 4, 2D inputmeshes 402 are each rectangular in shape with a particular height (e.g.,four vertices or three primitives) and a particular length (e.g., eightvertices or seven primitives). However, 2D input meshes may have anysuitable aspect ratio, size, and/or shape, and may use any suitable 2Darrangement of vertices and primitives as may suit a particularimplementation.

Rendering facility 102 may project a 2D input mesh within a virtual 3Dspace in any suitable manner. For example, rendering facility 102 mayproject a 2D input mesh along perspective rays associated the respectivevantage point to depths within the virtual 3D space based on the depthdata received by rendering facility 102. To illustrate, FIGS. 5-7 showexemplary projections of 2D input meshes 402 (e.g., 2D input meshes402-1 through 402-3) along perspective rays 502 (e.g., perspective rays502-1 through 502-3) within a virtual 3D space 504. In the examplesshown in FIGS. 5-7, virtual 3D space 504 and 2D input meshes 402 areshown in a cross-sectional view (e.g., a view of an x-z plane of virtual3D space 504 represented by an x-y-z coordinate system) with eachcircular dot of 2D input meshes 402 representing a vertex of a 2D inputmesh 402. While FIGS. 5-7 illustrate a cross-section of virtual 3D space504 to be rectangular, this is illustrative only. A virtual 3D space mayhave any suitable shape.

FIG. 5 shows a projection of 2D input mesh 402-1 along a plurality ofperspective rays 502-1 associated with vantage point 204-1 to depthswithin a virtual 3D space 504 that are represented by depth datacaptured by capture device 202-1 associated with vantage point 204-1 andreceived by rendering facility 102. The depth data may representsurfaces of one or more objects included in a captured 3D scene that arewithin a field of view of and detected by capture device 202-1 fromvantage point 204-1. FIG. 5 illustrates a dashed-line representation 506of object 208 (e.g., a cross-sectional view of a cube) and a dashed-linerepresentation 508 of another object (e.g., a cross-sectional view of abackwall) to represent positions of surfaces of objects within thecaptured 3D scene. The positions of the surfaces of the objects arerepresented as captured depth data, and rendering facility 102 may usethis depth data to project vertices of 2D input mesh 402-1 to 3Dcoordinates within virtual 3D space 504 that align with detectedpositions of surfaces of the objects within the captured 3D scene.

FIG. 6 similarly shows a projection of 2D input mesh 402-2 along aplurality of perspective rays 502-2 associated with vantage point 204-2to depths within virtual 3D space 504 that are represented by depth datacaptured by capture device 202-2 associated with vantage point 204-2 andreceived by rendering facility 102. FIG. 7 similarly shows a projectionof 2D input mesh 402-3 along a plurality of perspective rays 502-3associated with vantage point 204-3 to depths within virtual 3D space504 that are represented by depth data captured by capture device 202-3associated with vantage point 204-3 and received by rendering facility102.

In the examples shown in FIGS. 5-7, the projected 2D input meshes 402form partial 3D meshes 510 (e.g., partial 3D meshes 510-1 through 510-3)that virtually represent of at least part of surfaces of objectsincluded in the captured 3D scene. Partial 3D meshes 510 includevertices positioned at 3D coordinates in virtual 3D space 504. Thevertices positioned at the 3D coordinates define properties of theprimitives of the partial 3D meshes (e.g., size, position, orientation,and so forth) such that the vertices and primitives virtually represent,within virtual 3D space 504, locations of surfaces of objects in thecapture 3D scene.

The exemplary 2D input meshes 402 and corresponding partial 3D meshes510 shown in FIGS. 5-7 are provided for illustrative purposes only. Itis understood that in practice a cross-section of a 2D input mesh mayhave many more vertices than the eight vertices shown for each 2D inputmesh illustrated in FIGS. 5-7.

As shown in FIGS. 5-7, vertices of a partial 3D mesh may be projected todepths within virtual 3D space 504 that are associated with surfaces ofdifferent objects included in the captured 3D scene. For example, inFIG. 5, five of the eight cross-sectional vertices of partial 3D mesh510-1 are projected to depths associated with a surface of object 208 inthe captured 3D scene (represented as dashed-line representation 506 ofobject 208), while the other three of the cross-sectional vertices ofpartial 3D mesh 510-1 are projected to depths associated with, forexample, a surface of a background wall in the captured 3D scene(represented as dashed-line representation 508 of another object).Similarly, in FIGS. 6 and 7, vertices of partial 3D meshes are projectedto depths within virtual 3D space 504 that are associated with surfacesof objects included in the captured 3D scene.

The projection of a 2D input mesh within a virtual 3D space may producea partial 3D mesh that includes one or more primitives that may not beuseful or reliable to generate a partial virtual representation of acaptured scene or object in the virtual 3D space. Accordingly, renderingfacility 102 may perform one or more operations to identify and removenon-useful and/or unreliable primitives from a partial 3D mesh. Suchprimitives may be referred to as extraneous primitives and may includeany primitives that may not accurately represent a surface of an objectof a captured 3D scene with sufficient confidence, which confidencethreshold may be defined as may suit a particular application.

As an example, a partial 3D mesh may include one or more primitives thateach span a distance between a vertex projected to a surface of oneobject and a vertex projected to a surface of another object. Such aninter-object primitive may not accurately represent a surface of anobject in the captured 3D scene and as such may not be useful togenerate a partial virtual representation of the object. FIG. 5, forexample, shows that primitive 512 spans a distance from a vertexprojected to a surface of a cube object represented by dashed-linerepresentation 506 and a vertex projected to a surface of a backwallobject represented by dashed-line representation 508. Accordingly,primitive 512 may not represent a surface of the cube object 208 or ofthe backwall object. Accordingly, primitive 512 is not useful togenerate a partial virtual representation of object 208 or the backwallobject within a virtual 3D space.

Rendering facility 102 may perform one or more operations to identifyand remove extraneous primitives, such as primitive 512, from partial 3Dmeshes. Such operations may include rendering facility 102 identifying,for each partial 3D mesh, one or more of the primitives of the partial3D mesh as extraneous primitives and removing the identified extraneousprimitives from each partial 3D mesh. Rendering facility 102 mayidentify and remove extraneous primitives from a partial 3D mesh in anysuitable way, examples of which will now be described.

In certain examples, rendering facility 102 may identify a primitive ofa partial 3D mesh as an extraneous primitive based at least in part onthe primitive having a depth change that exceeds a predetermined depthchange threshold. The predetermined depth change threshold may bedetermined in advance in any suitable manner and may have any suitablevalue according to a particular implementation. In certain examples, thepredetermined depth change threshold may represent a maximum amount ofdepth change that a primitive may have and still be considered asrepresenting a surface of an object in a 3D scene.

FIG. 8 shows an example in which certain primitives included in partial3D mesh 510-2 have a depth change that exceeds a predetermined depthchange threshold. As shown in FIG. 8, partial 3D mesh 510-2 includesprimitives 802 (e.g., primitives 802-1 through 802-7). Of primitives802, primitives 802-2 and 802-6 may each have a respective depth change804 (e.g., depth changes 804-1 and 804-2) that exceeds a predetermineddepth change threshold. Because of depth changes 804, rendering facility102 may determine that primitives 802-2 and 802-6 do not accuratelyrepresent a surface of an object (e.g., object 208 or the backwallobject) in the captured 3D scene. Accordingly, rendering facility 102may identify primitives 802-2 and 802-6 as extraneous primitives andremove primitives 802-2 and 802-6 from partial 3D mesh 510-2 in anysuitable manner. Primitives 802-2 and 802-6 are shown in dotted lines inFIG. 8 to indicate that they have been removed by rendering facility102.

Additionally or alternatively, rendering facility 102 may identify aprimitive of a partial 3D mesh as an extraneous primitive based at leastin part on the primitive being out of range of a predetermined nearplane or a predetermined far plane. Rendering facility 102 may define apredetermined near plane and predetermined far plane in advance in anysuitable manner, such as based on metadata (e.g., a depth mapping rangeincluded in the metadata), a vantage point, and/or the depth data.

FIG. 9 shows an example in which primitives 802 of partial 3D mesh 510-2are illustrated in relation to a predetermined near plane 902 and apredetermined far plane 904. In the example shown in FIG. 9, renderingfacility 102 may determine that predetermined near plane 902 correspondsto either a depth value of a nearest depth point represented bydashed-line representation 506 or the depth value of the nearest depthpoint minus some predetermined offset. When all or a portion of aprimitive is located between predetermined near plane 902 and vantagepoint 204-2, rendering facility 102 may determine that the primitive isan extraneous primitive. Similarly, when all or a portion of a primitiveis located on or beyond far plane 904 (away from vantage point 204-2),rendering facility 102 may determine that the primitive is an extraneousprimitive. In the example shown in FIG. 9, at least a part of primitives802-2 and 802-6 are located beyond far plane 904. In addition,primitives 802-1 and 802-7 are located entirely beyond far plane 904.Accordingly, rendering facility 102 may determine that primitives 802-1,802-2, 802-6, and 802-7 are extraneous primitives and may remove themfrom partial 3D mesh 510-2 in any suitable manner. Primitives 802-1,802-2, 802-6, and 802-7 are shown in dotted lines in FIG. 9 to indicatethat they have been removed by rendering facility 102.

Additionally or alternatively, rendering facility 102 may identify aprimitive as an extraneous primitive based at least in part on theprimitive having a surface not facing the respective vantage pointassociated with a partial 3D mesh. Rendering facility 102 may determinewhether a primitive has a surface that does not face a respectivevantage point in any suitable manner. For example, rendering facility102 may determine that a primitive does not face a corresponding vantagepoint based on a relationship between a surface normal of the primitiveand a corresponding perspective ray from the corresponding vantagepoint. For example, rendering facility 102 may perform a cosine-theta ordot-product calculation to determine whether the surface of theprimitive faces a respective vantage point. If a result of thecosine-theta or dot-product calculation is a certain value, renderingfacility 102 may determine that the surface of the primitive faces awayfrom a vantage point (e.g., in a direction that is orthogonal to aperspective ray of the vantage point). Accordingly, rendering facility102 may identify such a primitive as an extraneous primitive.

FIG. 10 shows an example in which surface normal arrows are illustratedfor each primitive 802 of partial 3D mesh 510-2 to show the directionsthat surfaces of primitives 802 may face. In the example shown in FIG.10, rendering facility 102 may perform any suitable calculation todetermine that primitives 802-2 and 802-6 have surfaces that do not facevantage point 204-2. Accordingly, rendering facility 102 may identifyprimitives 802-2 and 802-6 as extraneous primitives and remove them frompartial 3D mesh 510-2 in any suitable manner.

In the example described above, rendering facility 102 may determinethat a surface of a primitive faces away from the corresponding vantagepoint if the angle between the corresponding perspective ray and thesurface normal for the surface is greater than or equal to an orthogonalangle. In certain other examples, rendering facility 102 may determinethat a surface of a primitive does not face a vantage point based on thesurface being within a predefined range of being orthogonal with respectto a perspective ray of a corresponding vantage point, as may suit aparticular implementation. Accordingly, rendering facility 102 mayidentify and remove a primitive having a surface that is angled relativeto a perspective ray of a vantage point by more than a predeterminedangle.

Rendering facility 102 may perform one or more operations, such as thosedescribed above, to generate a partial 3D mesh. In certain examples, agenerated partial 3D mesh may include all of the primitives produced byprojection of a 2D input mesh to depths defined by depth data for avantage point, as described above. In other examples, a generatedpartial 3D mesh may include less than all of the primitives produced byprojection of a 2D input mesh to depths defined by depth data for avantage point because one or more of the primitives may be identified asextraneous primitives and removed from the partial 3D mesh, leaving asubset of the primitives as the generated partial 3D mesh. For example,certain primitives 802 have been removed from partial 3D mesh 510-2shown in FIG. 9 such that only primitives 802-3 through 802-5 remain.Such operations may be performed for any number of partial 3D meshescorresponding to respective vantage points. Accordingly, renderingfacility 102 may generate partial 3D meshes, for all or a subset ofavailable vantage points, projected into a virtual 3D space to producepartial virtual representations of one or more objects in a captured 3Dscene (e.g., object 208 in 3D scene 206) that may be used by renderingfacility 102 to generate an image view of the partial virtualrepresentations in the virtual 3D space from an arbitrary viewpointwithin the virtual 3D space.

Rendering facility 102 may generate and use each partial 3D meshindividually, and each partial 3D mesh may represent only a partialvirtual representation of a 3D scene. For example, partial 3D mesh510-2, as illustrated in FIG. 9 to include primitives 802-3, 802-4, and802-5, only partially represents surfaces of object 208 in 3D scene 206.In this sense, partial 3D mesh 510-2 may be considered a partial mesh asopposed to a single, cohesive mesh for an entire virtual object orvirtual 3D world. While rendering facility 102 may generate and usepartial 3D meshes individually, because the partial 3D meshes areprojected into a common virtual 3D space, collectively the partial 3Dmeshes may represent all surfaces, or all relevant surfaces, of objectsin a captured 3D scene.

FIG. 11 shows a cross-sectional view of a plurality of partial 3D meshes1102 (e.g., partial 3D meshes 1102-1 through 1102-3) projected intovirtual 3D space 504. In the example shown in FIG. 11, the vertices ofeach partial 3D mesh 1102 are represented by a different shape indicatorthan the vertices of the other partial 3D meshes 1102. For example, thevertices of partial 3D mesh 1102-1 are represented by triangleindicators, the vertices of partial 3D mesh 1102-2 are represented bycircle indicators, and the vertices of partial 3D mesh 1102-3 arerepresented by square indicators. The vertices are represented bydifferent shapes in FIG. 11 to distinguish between the partial 3D meshes1102 and to illustrate that each partial 3D mesh 1102 is an individual,separate partial 3D mesh.

In addition, partial 3D meshes 1102 are shown in FIG. 11 as beingadjacent to one another and not lining up on top of one another forclarity of illustration. It is understood that in practice partial 3Dmeshes 1102 may line up on top of one another and with dashed-linerepresentation 506 of object 208 based on one or more of the verticesbeing projected to same depths within virtual 3D space 504 based on thedepth data representative of positions of surfaces of object 208 in thecaptured 3D scene 206. For example, while partial 3D meshes 1102-1 and1102-3 are shown to be positioned slightly in front of partial 3D mesh1102-2 and dashed-line representation 506 of object 208 in FIG. 11, thisis for clarity of illustration. In practice, partial 3D meshes 1102-1and 1102-3 will align more closely with partial 3D mesh 1102-2 anddashed-line representation 506 of object 208.

FIG. 11 illustrates that individual partial 3D meshes may be projectedinto a common virtual 3D space such that the individual partial 3Dmeshes are superimposed and may have overlapping sections that sharecommon spatial positions in the virtual 3D space. Sections of partial 3Dmeshes may be considered overlapping when the sections share commonspatial positions in the virtual 3D space. For example, a primitive 1104of partial 3D mesh 1102-2 and a section 1106 of partial 3D mesh 1102-1may be overlapping when aligned with common spatial positionsrepresenting a surface of object 208 in virtual 3D space 504.

FIG. 11 illustrates three partial 3D meshes 1102-1 through 1102-3projected into virtual 3D space. Partial 3D meshes 1102-1 through 1102-3may be partial 3D meshes generated by rendering facility 102 in any ofthe ways described above based on vantage points 204-1 through 204-3 ofcapture devices 202-1 through 202-3 in the configuration 200 shown inFIG. 2. However, any suitable number of partial 3D meshes for anysuitable of configuration of capture devices having vantage pointsrelative to a 3D scene may be projected into a common virtual 3D spaceto produce partial virtual representations of the 3D scene in thevirtual 3D space. For example, for a configuration in which additionalcapture devices having additional vantage points relative to object 208in a 3D scene, rendering facility 102 may generate additional partial 3Dmeshes, for the additional vantage points, projected into virtual 3Dspace 504 to virtually represent additional surfaces of object 208 invirtual 3D space 504. To illustrate, FIG. 12 shows a cross-sectionalview of a plurality of partial 3D meshes 1202 (e.g., partial 3D meshes1202-1 through 1202-6) projected into virtual 3D space 504. FIG. 12 issimilar to FIG. 11 except that FIG. 12 shows six partial 3D meshes 1202associated with six vantage points projected into virtual 3D space 504.

In certain examples, rendering facility 102 may generate partial 3Dmeshes projected into a common virtual 3D space for all availablevantage points relative to a captured 3D space. In other examples,rendering facility 102 may generate partial 3D meshes projected into acommon virtual 3D space for only a subset of all available vantagepoints relative to a captured 3D space. Rendering facility 102 maygenerate partial 3D meshes for only a subset of available vantage pointsbecause rendering facility 102 receives data for only a subset ofavailable vantage point or because rendering facility 102 may select andprocess data for a subset of the most preferred vantage points, such asthe vantage points that provide the most reliable data for an arbitraryviewpoint in the virtual 3D space.

Rendering facility 102 may use generated partial 3D meshes projectedinto a virtual 3D space to generate an image view of the virtual 3Dspace from an arbitrary viewpoint within the virtual 3D space. Becausethe partial 3D meshes projected into the virtual 3D space producepartial virtual representations of a captured 3D scene (i.e., partialvirtual reconstructions of one or more objects in the 3D scene), asdescribed above, the image view may represent a perspective view of thepartial virtual representations of the 3D scene from any arbitraryviewpoint within the virtual 3D space. The generated image view may berepresented in any suitable way, including as data (e.g., datarepresentative of fragments) mapped (e.g., rasterized) from 3Dcoordinates in virtual 3D space 504 to a set of 2D image coordinates inan image plane such that the data may be used to generate and outputdisplay screen data (e.g., pixel data) representative of the image viewfor display on a 2D display screen.

Rendering facility 102 may generate an image view of the virtual 3Dspace from an arbitrary viewpoint within the virtual 3D space byaccumulating partial 3D meshes projected into the virtual 3D space andblending color samples for the partial 3D meshes to form the image viewof the virtual 3D space. In certain examples, rendering facility 102 mayaccumulate the partial 3D meshes in a frame buffer of a GPU andadditively blend color samples for the partial 3D meshes in the framebuffer of the GPU. In certain examples, the blending of color samplesfor partial 3D meshes may include rendering facility 102 determiningblend weights for primitives of the partial 3D meshes, weighting thecolor samples for the partial 3D meshes based on the determined blendweights for the primitives of the partial 3D meshes, and determiningweighted averages of the weighted color samples for superimposedprimitives included in overlapping sections of the partial 3D meshes. Incertain examples, rendering facility 102 may determine an accumulationregion and accumulate and blend only primitives or fragments that arelocated within the accumulation region. Examples of such operations forgenerating an image view of a virtual 3D space from an arbitraryviewpoint within the virtual 3D space based on partial 3D meshesprojected into the virtual 3D space will now be described.

FIG. 13 illustrates a cross-sectional view (e.g., a view of an x-z planeof virtual 3D space 504) of partial 3D meshes 1202 projected intovirtual 3D space 504 and that may be used by rendering facility 102 togenerate an image view 1302 of virtual 3D space 504 from an arbitraryviewpoint 1304 in virtual 3D space 504. In FIG. 13, image view 1302 isrepresented as a cross-sectional view of a 2D image plane in virtual 3Dspace 504 and onto which a perspective view of virtual 3D space 504 maybe projected from arbitrary viewpoint 1304.

To generate image view 1302, rendering facility 102 may accumulate datarepresentative of partial 3D meshes 1202 projected in virtual 3D space504 into a frame buffer in which data representative of image view 1302is written. The frame buffer may be any buffer, such as a buffer of aGPU, capable of storing data samples from accumulated partial 3D meshes.In certain examples, the frame buffer may be a high-precision framebuffer such as a frame buffer configured to store 16-bit float type dataor any other data type suitable for high dynamic range video or imageformat. In other examples, the frame buffer may be configured to storeany other data type that may be used for any suitable image and/or videoformat.

Rendering facility 102 may accumulate partial 3D meshes 1202 into theframe buffer in any suitable way, including by sampling data for thepartial 3D meshes to select which data to write to the frame buffer,writing selected data to the frame buffer, and not writing unselecteddata to the frame buffer. To this end, rendering facility 102 mayperform one or more operations to select which data to write or notwrite to the frame buffer, such as operations for determining occlusionwithin virtual 3D space 504, determining directions that surfacesrepresented by partial 3D meshes 1202 face, depth testing, and so forth.Using such operations, rendering facility 102 may determine which datafrom partial 3D meshes 1202 virtually represents a surface of an objectthat is visible from arbitrary viewpoint 1304 and may select and writethat data to the frame buffer.

To illustrate, from arbitrary viewpoint 1304 in FIG. 13, primitives ofpartial 3D meshes 1202 that virtually represent a front-right surface ofobject 208 in virtual 3D space are visible and thus included in imageview 1302. As shown, such primitives include one primitive of partial 3Dmesh 1202-2 (labeled as primitive 1306), all primitives of partial 3Dmesh 1202-3 (not labeled with individual reference numbers), and oneprimitive of partial 3D mesh 1202-4 (labeled as primitive 1308). In thisexample, rendering facility 102 may perform operations to identify theseprimitives as visible from arbitrary viewpoint 1304 and to write datarepresentative of these primitives to the frame buffer. Renderingfacility 102 may also perform operations to identify other primitives asnot visible from arbitrary viewpoint 1304 and to not write datarepresentative of these other primitive to the frame buffer, which mayinclude never writing the data representative of non-visible primitivesto the frame buffer and/or overwriting data representative ofnon-visible primitives that has already been written to the frame bufferwith data representative of the primitives identified to be visible fromarbitrary viewpoint 1304, depending on an order in which renderingfacility 102 samples (e.g., depth tests) data from partial 3D meshes1202.

In certain examples, rendering facility 102 may sample data from allpartial 3D meshes 1202 projected into virtual 3D space 504 in any orderand write data representative of selected primitives to the frame buffersuch that after all partial 3D meshes 1202 have been sampled, the framebuffer includes data only for primitives of partial 3D meshes that arevisible from arbitrary viewpoint 1304. In other examples, renderingfacility 102 may select a subset of partial 3D meshes 1202 projected invirtual 3D space 504 that are most relevant to arbitrary viewpoint 1304,based on any suitable defined criteria and operations, and sample dataonly for the selected partial 3D meshes 1202.

Because rendering facility 102 may accumulate partial 3D meshes 1202that have overlapping sections associated with common 3D coordinates invirtual 3D space 504, rendering facility 102 may select multipleprimitives or samples, from multiple partial 3D meshes 1202, for acommon 3D coordinate in virtual 3D space 504 to be mapped to a common 2Dcoordinate of the image view 1302 as represented in the frame buffer.For such overlapping sections of partial 3D meshes 1202, renderingfacility 102 may perform one or more operations to accumulate and blendcolor samples for the overlapping sections of partial 3D meshes 1202 inthe frame buffer. In certain examples, rendering facility 102 usesadditive blending to blend the color samples such that samples ofpartial 3D meshes 1202 may be written to the frame buffer as the samplesare sampled and selected to be written, for any order of processing ofpartial 3D meshes 1202.

An example of rendering facility 102 accumulating and blending samplesfrom partial 3D meshes for a single 2D coordinate point of image view1302 will now be described. FIG. 13 shows an exemplary perspective ray1310 projected from arbitrary viewpoint 1304 into virtual 3D space 504.As shown, perspective ray 1310 intersects image view 1302 at a 3Dcoordinate point 1312 in virtual 3D space 504, partial 3D mesh 1202-3 ata 3D coordinate point 1314 in virtual 3D space 504, partial 3D mesh1202-2 at a 3D coordinate point 1316 in virtual 3D space 504, andpartial 3D mesh 1202-6 at a 3D coordinate point 1318. Although 3Dcoordinate points 1314 and 1316 are shown at different locations in FIG.13 because partial 3D meshes 1202-2 and 1202-3 are not aligned on top ofone another for clarity of illustration as described above, 3Dcoordinate points 1314 and 1316 are considered to be the same 3Dcoordinate point or within a predetermined threshold distance of oneanother so as to be treated as the same 3D coordinate point. 3Dcoordinate point 1312 on the image plane of image view 1302 correspondsto a 2D coordinate point of the image view 1302 within a set of 2Dcoordinates used to represent image view 1302 and to which 3D coordinatepoints 1314, 1316, and 1318 may be mapped by rendering facility 1302when accumulating data from partial 3D meshes 1202 in a frame buffer.

As an example, rendering facility 102 may sample 3D coordinate point1314 of partial 3D mesh 1202-3, select the 3D coordinate point 1314 tobe written to the frame buffer, and map and write data for the 3Dcoordinate point 1314 to the 2D coordinate point for the image view 1302that corresponds to 3D coordinate point 1312 on the image plane of imageview 1302. Rendering facility 102 may also sample 3D coordinate point1318 of partial 3D mesh 1202-6, not select the 3D coordinate point 1318to be written to the frame buffer because 3D coordinate point 1318 isoccluded by 3D coordinate point 1314 already mapped and written to the2D coordinate point for the image view 1302 in the frame buffer (whichocclusion may be determined by depth testing or in any other suitableway), and not write data for the 3D coordinate point 1318 to the 2Dcoordinate point for the image view 1302. Rendering facility 102 mayalso sample 3D coordinate point 1316 of partial 3D mesh 1202-2 andselect the 3D coordinate point 1316 to be written to the frame bufferbecause 3D coordinate point 1316 is not occluded by 3D coordinate point1314 because they represent a common 3D point. Because 3D coordinatepoint 1316 is the same as 3D coordinate point 1314, as described above,rendering facility 102 may map and write data for the 3D coordinatepoint 1316 to the 2D coordinate point for the image view 1302 in a waythat blends data for the 3D coordinate points 1314 and 1316 such thatthe frame buffer will include data for the 2D coordinate point thatrepresents a blending of data for 3D coordinate points 1314 and 1316.For example, rendering facility 102 may blend color samples for 3Dcoordinate points 1314 and 1316 in the frame buffer.

In certain examples, rendering facility 102 may blend color samples forpartial 3D meshes by additively blending color samples for common 3Dcoordinate points on overlapping sections of partial 3D meshes in aframe buffer as rendering facility 102 processes samples. Continuing theexample shown in FIG. 13, rendering facility 102 may additively blendcolor samples for common 3D coordinate points 1314 and 1316 in the framebuffer for a 2D coordinate point in image view 1302 to which 3Dcoordinate points 1314 and 1316 are mapped. To illustrate, renderingfacility 102 may sample 3D coordinate point 1314 of partial 3D mesh1202-3, select the 3D coordinate point 1314 to be written to the framebuffer, and map and write data for the 3D coordinate point 1314 to the2D coordinate point for image view 1302. This may include writing colorsample data associated with 3D coordinate point 1314 to the framebuffer. For instance, the color sample data for 3D coordinate point inRGB color model value may be (255, 0, 0), and rendering facility 102 maywrite this value to the frame buffer (e.g., as part of fragment data)for the 2D coordinate point of image view 1302. Rendering facility 102may also sample 3D coordinate point 1316 of partial 3D mesh 1202-2 andselect the 3D coordinate point 1316 to be written to the frame buffer asdescribed above. Because 3D coordinate point 1316 is the same as 3Dcoordinate point 1314, writing data for the 3D coordinate point 1316 tothe 2D coordinate point for the image view 1302 may include writingcolor sample data associated with 3D coordinate point 1316 to the framebuffer in a way that blends color sample data for the 3D coordinatepoints 1314 and 1316 such that the frame buffer will include colorsample data for the 2D coordinate point that represents a blending ofcolor samples for 3D coordinate points 1314 and 1316. For instance, thecolor sample data for 3D coordinate point in RGB color model value maybe (205, 0, 0), and rendering facility 102 may additively blend thisvalue to the value RGB value for 3D coordinate point 1314 already storedin the frame buffer for the 2D coordinate point of image view 1302. Incertain examples, the additive blending may include determining anaverage of the RGB color values for the 3D coordinate points 1314 and1316 and writing the determined average (e.g., RGB value (225, 0, 0)) tothe frame buffer for the 2D coordinate point of image view 1302.

In certain examples, rendering facility 102 may use weighted blending toblend color samples for partial 3D meshes. For example, the blending ofcolor samples for partial 3D meshes may include rendering facility 102determining blend weights for primitives of the partial 3D meshes,weighting the color samples for the partial 3D meshes based on thedetermined blend weights for the primitives of the partial 3D meshes,and determining weighted averages of the weighted color samples forsuperimposed primitives included in overlapping sections of the partial3D meshes.

Rendering facility 102 may determine blend weights for primitives ofpartial 3D meshes in any suitable way. In certain examples, renderingfacility 102 may determine a blend weight of a primitive based on adirection that a surface of the primitive faces in relation to a vantagepoint from which data associated with the primitive is captured, such asa relationship (e.g., an angle) between a surface normal of the surfaceof the primitive and a perspective ray projected from the vantage pointand intersecting the primitive. Such a relationship may represent alevel of confidence in or quality of the surface data for the primitivebased on an angle of the surface of the primitive relative to thevantage point from which the surface data is captured. For example, asmaller angle between the surface normal of the primitive and aperspective ray projected from a capture vantage point may be indicativeof a higher confidence level in the surface data for the primitive thanwould be indicated by a larger angle between the surface normal of theprimitive and a perspective ray projected from a capture vantage point.The smaller angle is indicative of the surface of the primitive moredirectly facing the capture vantage point compared to a larger anglethat is indicative of the surface of the primitive less directly facingthe capture vantage point. Accordingly, based on angular relationshipsbetween primitive surfaces and capture vantage points, renderingfacility 102 may determine a higher blend weight for a first primitivehaving a surface that more directly faces a capture vantage point and alower blend weight for a second primitive having a surface that lessdirectly faces a capture vantage point, in order to give more weight tosurface data associated with the first primitive than is given tosurface data associated with the second primitive. In this or a similarmanner, rendering facility 102 may control how primitives influenceblending operations based on the level of confidence in the surface datafor the primitives, such as by reducing the influence of angled surfacesin blending operations.

Continuing with the example illustrated in FIG. 13, for example, 3Dcoordinate point 1314 may be on a primitive of partial 3D mesh 1202-3that has a surface that more directly faces a vantage point (e.g.,vantage point 204-3) from which surface data associated with partial 3Dmesh 1202-3 is captured, than does the surface of the primitive ofpartial 3D mesh 1202-2 on which 3D coordinate point 1316 is located withrespect to a vantage point (e.g., vantage point 204-2) from whichsurface data associated with partial 3D mesh 1202-2 is captured.Accordingly, rendering facility 102 may determine a relatively higherblend weight for the primitive associated with 3D coordinate point 1314on partial 3D mesh 1202-3 and a relatively lower blend weight for theprimitive associated with 3D coordinate point 1316 on partial 3D mesh1202-2, which blend weights may be used by rendering facility 102 toincrease the influence of the primitive associated with 3D coordinatepoint 1314 and decrease the influence of the primitive associated with3D coordinate point 1316 when color samples for the 3D coordinate points1314 and 1316 are blended.

Rendering facility 102 may use the determined blend weights forprimitives of partial 3D meshes to weight the surface data (e.g., colorsamples) for the partial 3D meshes and determine weighted averages ofthe weighted surface data for superimposed primitives included inoverlapping sections of the partial 3D meshes. For example, renderingfacility 102 may use blend weights for primitives of partial 3D meshes1202-3 and 1202-2 to weight color samples for 3D coordinate points 1314and 1316 and to determine a weighted average of the weighted colorsamples. For instance, rendering facility 102 may multiply a colorsample for 3D coordinate point 1314 by a first blend weight determinedfor the primitive of partial 3D mesh 1202-3 that includes 3D coordinatepoint 1314, multiply a color sample for 3D coordinate point 1316 by asecond blend weight determined for the primitive of partial 3D mesh1202-2 that includes 3D coordinate point 1316, add the products of thetwo multiplications, and divide the sum of the products by a sum totalof the blend weights to determine a weighted average of the colorsamples for 3D coordinate points 1314 and 1316 of superimposedprimitives included in overlapping sections of partial 3D meshes 1202-3and 1202-2. Rendering facility 102 may write the determined weightedaverage of the color samples to the frame buffer for the corresponding2D coordinate point of image view 1302.

Weighting color samples in this or a similar manner gives relativelymore weight to a color sample associated with a more directly capturedsurface and relatively less weight to a color sample associated with aless directly captured surface. For example, if a weight of 0.7 isdetermined for a color sample for 3D coordinate point 1314 having RGBvalue (255, 0, 0) and a weight of 0.3 is determined for a color samplefor 3D coordinate point 1316 having RGB value (205, 0, 0), renderingfacility 102 may determine a weighted average color sample RGB value of(240, 0, 0) for the blended color samples, which weighted average colorsample value may be written in the frame buffer for the corresponding 2Dcoordinate point as a result of blending the color samples in the framebuffer.

As used herein, writing of data to a frame buffer may include writingdata to any suitable frame buffer or configuration of frame buffers thatstore data for a 2D image view of a virtual 3D space. The data writtento the frame buffer may include any data helpful as display data and/orfor determining display data (e.g., pixel data) that may be output to adisplay device and used to display a 2D image view on a 2D displayscreen or a set of multiple 2D display screens.

FIG. 14 illustrates an exemplary configuration 1400 of frame buffersthat includes an accumulation buffer 1402 and an output buffer 1404. Incertain examples, rendering facility 102 may use accumulation buffer1402 to accumulate weighted data for partial 3D meshes. To this end,accumulation buffer 1402 may include a set of data for a weightedfragment in which RGB color model values are multiplied by a blendweight “w.” Accumulation buffer 1402 may also include datarepresentative of the blend weight “w,” which blend weight may bedetermined by rendering facility 102 as described herein. Renderingfacility 102 may accumulate partial 3D meshes in accumulation buffer asdescribed herein, including by adding all weighted fragments for acommon 2D coordinate point of an image view. Once all samples have beenaccumulated in accumulation buffer 1402, rendering facility 102 maydivide the sum of the weighted fragments by the sum of blend weights inthe accumulation buffer 1402 to determine a weighted average of thefragments for the common 2D coordinate point. Rendering facility 102 mayoutput (e.g., write) the determined weighted average of the fragments tooutput buffer 1404 (e.g., in RGBA color model format as shown or in anyother suitable format), which may be used to provide a display of the 2Dimage view on a display screen for viewing by a user associated with thedisplay screen.

In FIG. 14, accumulation buffer 1402 is shown to include a set of datafor a single 3D coordinate point in virtual 3D space 504 that is mappedto a 2D coordinate point on image view 1302. It will be understood thataccumulation buffer 1402 may include additional sets of data for one ormore other 3D coordinate points in virtual 3D space 504 that are mappedto the same 2D coordinate point on image view 1302 to allow weightedaccumulation and blending of the sets of data for all 3D coordinatepoints in virtual 3D space 504 that are mapped to the same 2D coordinatepoint on image view 1302.

In FIG. 14, accumulation buffer 1402 and output buffer 1404 are eachshown to include a set of data for a single 2D coordinate point of animage view (e.g., image view 1302). It will be understood thataccumulation buffer 1402 and output buffer 1404 may each include theillustrated set of data (and additional data not illustrated in FIG. 14,in certain implementations) for each 2D coordinate point in a 2D arrayof 2D coordinate points that represent a 2D image view. In certainexamples, accumulation buffer 1402 may be a higher-precision buffer thandisplay precision and/or output buffer 1404 to allow for accurateaccumulation calculations on data in accumulation buffer 1402. Forexample, accumulation buffer 1402 may store data in a 16-bit float,16-bit normalized, or other high dynamic range video or image data typeformat, and output buffer 1404 may store data in an 8-bit low dynamicrange video or image data type format. Accumulation buffer 1402 mayfunction as an intermediary buffer in which data for multiple partial 3Dmeshes may be accumulated and then output to output buffer 1404 asdescribed herein.

In certain examples, rendering facility 102 may determine anaccumulation region and accumulate and/or blend only primitives orfragments that are located within the accumulation region. Renderingfacility 102 may determine the accumulation region at any suitable time(e.g., as part of a depth pre-pass) and in any suitable manner, such asbased on metadata (e.g., a depth mapping range included in themetadata), an arbitrary viewpoint within the virtual 3D space, and/orthe depth data for partial 3D meshes projected into the virtual 3Dspace. For example, rendering facility 102 may determine, based on thedepth data for partial 3D meshes 1202 projected into virtual 3D space504, that an accumulation region begins at a depth value of a nearestdepth point of projected partial 3D meshes 1202 along perspective froman arbitrary viewpoint within virtual 3D space 504. Rendering facility102 may add a predetermined depth offset value to the determined depthvalue of the nearest depth point of projected partial 3D meshes 1202 todetermine a far end of the accumulation range. The accumulation regionmay extend from the determined depth value of the nearest depth point ina direction along perspective and away from the arbitrary viewpoint to adepth value of a far depth point that has a depth value equal to the sumof the nearest depth point and the predetermined depth offset value. Ifthe nearest depth point is represented as z0 and the predetermined depthoffset is represented as dz, then the far depth point may be representedas z1, where z1=z0+dz, and the accumulation range is [z0, z1] where thedepth z is oriented along perspective from the arbitrary viewpoint.

An accumulation region may have any size (i.e., range of depths within avirtual 3D space) as may suit a particular implementation. The size ofthe accumulation region may be determined by the value of thepredetermined depth offset, which may be defined in advance as may suita particular implementation. For example, the depth offset may bedefined to provide a desired level of precision for a 3D scene and/orvirtual 3D space.

The depth offset may be used by rendering facility 102 to push back thedepth value of each nearest sample of projected partial 3D meshes alongperspective from an arbitrary viewpoint in virtual 3D space 504. Toillustrate, FIG. 15 shows an exemplary cross-sectional view ofdashed-line representation 506 of object 208 in relation to an arbitraryviewpoint 1502 within virtual 3D space 504. In FIG. 15, the front edgesof dashed-line representation 506 of object 208 are the nearest depthsamples of projected partial 3D meshes to arbitrary viewpoint 1502 andare represented as solid lines 1504-1 and 1504-2. The nearest samplesare pushed back along perspective by a depth offset, which depth offsetis represented as arrows 1506. By pushing back the nearest samples bythe depth offset, rendering facility 102 may determine an accumulationregion along each perspective from the arbitrary viewpoint 1502.Examples of accumulation regions 1508 are shown in FIG. 15. In FIG. 15,dashed-line representation 1510 of object 208 represents object invirtual 3D space 504 after rendering facility 102 has performed thedepth offset. The front edges of dashed-line representation 1510 ofobject 208 are the nearest depth samples of projected partial 3D meshesto arbitrary viewpoint 1502 after depth offset and are represented assolid lines 1512-1 and 1512-2. Because rendering facility 102 pushesback the nearest samples along perspective from arbitrary viewpoint1502, the depth offset may be transparent to a user.

As shown in FIG. 15, accumulation regions 1508 may each span a distance,along perspective from arbitrary viewpoint 1502, from a nearest depthsample value to a depth offset value. Along the perspective fromarbitrary viewpoint 1502 associated with each accumulation region 1508,rendering facility 102 may accumulate and blend only primitives orfragments (e.g., color samples for the primitives or fragments) that arelocated within the respective accumulation region 1508. Accordingly,rendering facility 102 may ignore samples outside of (e.g., beyond) eachaccumulation region 1508, which may conserve computing resources and/orallow for fast processing of data. Each accumulation region (e.g., eachaccumulation region 1508) may be considered an accumulation region withrespect to an arbitrary viewpoint in a virtual 3D space, or acombination of accumulation regions (e.g., accumulation regions 1508) ina virtual 3D space may be considered an accumulation region for thevirtual 3D space with respect to an arbitrary viewpoint.

Projected partial 3D meshes may be superimposed and overlap within avirtual 3D space (e.g., as illustrated in FIGS. 11 and 12). Accordingly,the partial 3D meshes may have different depth values along perspectivefrom an arbitrary viewpoint. Such differences in depth values may occurbecause of encoding errors, capture errors, and/or any other errorassociated with the depth data received by rendering facility 102.Rendering facility 102 may correct and/or compensate for suchdifferences in depth values by using a depth offset from a front-mostsample to determine an accumulation region and accumulating sampleswithin a determined accumulation region, which allows for samplesassociated with different depth values within the accumulation region tobe sampled and blended together to contribute to an image view from thearbitrary viewpoint.

In certain examples, system 100 may be incorporated in a media playerdevice that may receive 2D color data, depth data, and metadata for acaptured 3D scene and use the received data to render virtual realitycontent, as described herein, for presentation to a user of the mediaplayer device such that the user of the media player device mayexperience a virtual reconstruction of the 3D scene from a dynamicallyselected viewpoint within the virtual reconstruction of the 3D scene.FIG. 16 shows an exemplary implementation 1600 of system 100 in whichsystem 100 is incorporated in a media player device 1602 communicativelycoupled to a virtual reality media provider system 1604. Media playerdevice 1602 may be communicatively coupled to virtual reality mediaprovider system 1604 in any suitable manner. In the example shown inFIG. 16, media player device 1602 is communicatively coupled to virtualreality media provider system 1604 by way of a network 1606. However,media player device 1602 may additionally or alternatively becommunicatively coupled directly to virtual reality media providersystem 1604.

Network 1606 may include any provider-specific wired or wireless network(e.g., a cable or satellite carrier network or a mobile telephonenetwork), the Internet, a wide area network, or any other suitablenetwork. Data may flow between media player device 1602 and virtualreality media provider system 1604 using any communication technologies,devices, media, and protocols as may serve a particular implementation.For example, media player device 1602 and virtual reality media providersystem 1604 may communicate using any suitable communicationtechnologies, devices, media, and/or protocols supportive of datacommunications, including, but not limited to, socket connections,Ethernet, data bus technologies, data transmission media, communicationdevices, media streaming technologies (e.g., video streamingtechnologies), Moving Picture Experts Group (“MPEG”) protocols,Transmission Control Protocol (“TCP”), Internet Protocol (“IP”), FileTransfer Protocol (“FTP”), Telnet, Hypertext Transfer Protocol (“HTTP”),HTTPS, Session Initiation Protocol (“SIP”), Simple Object AccessProtocol (“SOAP”), Extensible Mark-up Language (“XML”) and variationsthereof, Real-Time Transport Protocol (“RTP”), User Datagram Protocol(“UDP”), Global System for Mobile Communications (“GSM”) technologies,Code Division Multiple Access (“CDMA”) technologies, Evolution DataOptimized Protocol (“EVDO”), 4G Long Term Evolution (“LTE”), WiMax, TimeDivision Multiple Access (“TDMA”) technologies, radio frequency (“RF”)signaling technologies, wireless communication technologies (e.g.,BLUETOOTH, Wi-Fi, etc.), in-band and out-of-band signaling technologies,and other suitable communications technologies. While only one network1606 is shown to interconnect virtual reality media provider system 1604and media player device 1602 in FIG. 2, it will be recognized that thesedevices and systems may intercommunicate by way of multiple and/ordifferent interconnected networks as may serve a particularimplementation.

Virtual reality media provider system 1604 may include any hardwareand/or software (e.g., computing systems, software programs, etc.) usedfor processing 2D captured imagery and metadata. Virtual reality mediaprovider system 1604 may be located at a single location or distributedbetween multiple devices (e.g., servers) and/or multiple locations asmay serve a particular implementation.

Virtual reality media provider system 1604 may provide a data pipelinefor delivery of 2D captured imagery and metadata in which the 2Dcaptured imagery and metadata flows from capture devices (e.g., capturedevices 202) to virtual reality media provider system 1604 in anysuitable manner (e.g., directly or through network 1606) and fromvirtual reality media provider system 1604 to media player device 1602by way of network 1606. For example, virtual reality media providersystem 1604 may process acquired 2D captured imagery and metadata togenerate and provide the 2D captured imagery and metadata in at leastone transport stream for streaming to media player device 1602 by way ofnetwork 1606.

In certain examples, virtual reality media provider system 1604 maygenerate and provide 2D captured imagery and metadata to a media playerdevice such as media player device 1602 in any of the ways described inU.S. patent application Ser. No. 15/610,575 titled “Methods and Systemsfor Using 2D Captured Imagery of a Scene to Provide Virtual RealityContent” and filed the same day as the present application, whichapplication is hereby incorporated by reference herein in its entirety.

In certain examples, virtual reality media provider system 1604 maystream a transport stream to media player device 1602 through anysuitable communication channel. Alternatively, virtual reality mediaprovider system 1604 may provide the transport stream to a contentdelivery network (“CDN”) (e.g., a third-party CDN) for streaming tomedia player device 1602. In certain examples, virtual reality mediaprovider system 1604 may include metadata in the transport stream (e.g.,together with 2D captured imagery data (i.e., surface data)) and providethe transport stream for streaming such that the metadata is streamed tomedia player device 1602 as part of the transport stream. In otherexamples, virtual reality media provider system 1604 may provide themetadata and the transport stream separately for streaming such that themetadata may be streamed separately from the transport stream.

As used herein, “a transport stream” may refer to a single transportstream into which color video data streams and depth video data streamsmay be packaged, or to multiple transport streams into which color videodata streams and depth video data streams may be packaged. Thus, incertain examples, a transport stream may refer to a single transportstream that carries video data streams for a captured scene (e.g., onecolor video data stream for each 3D capture device and one depth videodata stream for each 3D capture device) as well as any metadata or othersuitable data that may be included for transport in a particularimplementation. In other examples, a transport stream may refer to aplurality of transport streams that collectively carry video datastreams for a captured scene (e.g., one color video data stream for each3D capture device and one depth video data stream for each 3D capturedevice) as well as any metadata or other suitable data that may beincluded for transport in a particular implementation.

A transport stream may include any type of transport stream (e.g., anMPEG transport stream or other type of transport stream) suitable fortransporting video and/or other data from virtual reality media providersystem 1604 to media player device 1602. A transport stream may beconfigured in accordance with any suitable data format, containerformat, and/or transport protocol as may suit a particularimplementation.

FIG. 17 shows an example of a transport stream 1702 in which a colorvideo data stream 1704-1, a depth video data stream 1706-1, and ametadata stream 1708-1 are provided as streams from capture device202-1, a color video data stream 1704-2, a depth video data stream1706-2, and a metadata stream 1708-2 are provided as streams fromcapture device 202-2, and a color video data stream 1704-n, a depthvideo data stream 1706-n, and a metadata stream 1708-n are provided asstreams from capture device 202-n.

The 2D captured imagery data transmitted or otherwise provided in atransport stream (e.g., transport stream 1702) to media player device1602 may be provided as separate color data streams and depth datastreams in any suitable manner. To illustrate, FIG. 18 shows exemplarysurface data 1802 that may be acquired by virtual reality media providersystem 1604 from, for example, capture device 202-2. As shown in FIG.18, surface data 1802 may include a series of sequential 2D color datacaptures 1804 (e.g. frames of a color data stream) of 3D scene 206captured by the capture device 202-2 and a series of sequential 2D depthdata captures 1806 (e.g., frames of a depth data stream) of scene 206captured by capture device 202-2. The series of sequential 2D color datacaptures 1804 may represent color data of object 208 in 3D scene 206from vantage point 204-2 of capture device 202-2 during a particulartemporal sequence 1808 (e.g., a particular period of real time, aparticular virtual timeline associated with 3D scene 206, etc.). Theseries of sequential 2D depth data captures 1806 may represent depthdata of object 208 in 3D scene 206 from vantage point 204-2 of capturedevice 202-2 during the particular temporal sequence 1808.

FIG. 18 also shows an exemplary 2D color data capture 1810 included inthe series of sequential 2D color data captures 1804 and an exemplary 2Ddepth data capture 1812 included in the series of sequential 2D depthdata captures. 2D color data capture 1810 may include color data, whichmay represent a view of 3D scene 206 (including color data from thesurfaces of object 208) visible from vantage point 204-2. While thecolor data is illustrated as an image in FIG. 18, it will be understoodthat the color data may be captured, encoded, formatted, transmitted,and represented in any suitable form. For example, the color data may bedigital data that is formatted according to a standard video encodingprotocol, a standard image format, or the like. The color data mayrepresent a color image (e.g., similar to a color photograph) of theobjects included within the 3D scene as viewed from virtual vantagepoint 204-2. Alternatively, the color data may be a grayscale imagerepresentative of the objects (e.g., similar to a black and whitephotograph).

2D depth data capture 1812 may include depth data for the surfaces ofobject 208 from a point in space associated with vantage point 204-2.Like the color data, the depth data represented in 2D depth data capture412 may depict object 208 within 3D scene 206 from the perspective ofvantage point 204-2. However, rather than representing the visibleappearance of object 208 (i.e., representing in color or grayscale howlight interacts with the surfaces of object 208), the depth data mayrepresent the depth (i.e., the distance or position) of points onsurfaces of object 208 (e.g., as well as other objects within 3D scene206) relative to the position of vantage point 204-2. As with the colordata, the depth data may be captured, encoded, formatted, transmitted,and represented in any suitable form. For example, as shown, the depthdata may be represented by grayscale image data (e.g., six or eight bitsfor each pixel represented within the depth data). However, rather thanrepresenting how visible light reflects from the surfaces of object 208(i.e., as represented in the color data), the grayscale image of thedepth data may represent, for each pixel in the image, how far away thepoint represented by that pixel is from the position of vantage point204-2. For example, points that are closer to vantage point 204-2 may berepresented with values that represent darker shades of gray (e.g.,binary values closer to 0b111111 in the case of a six-bit implementationwhere 0b111111 represents black). Conversely, points that are fartheraway from vantage point 204-2 may be represented with values thatrepresent lighter shades of gray (e.g., binary values closer to 0b000000in the case of the six-bit implementation where 0b000000 representswhite).

The metadata provided by virtual reality content provider system 1604may be formatted in any suitable manner. In certain examples, themetadata may be formatted into a series of sequential metadata captures.Each metadata capture included in the series of sequential projectionmetadata captures may include metadata associated with a particularcapture device at a particular point in time in relation to the 3Dscene. In addition, each metadata capture may be synchronized with acorresponding 2D color data capture and a corresponding 2D depth datacapture and, as such, may be referred to as metadata for the 2D colordata and depth data (e.g., as a metadata capture or frame forsynchronized 2D color and depth data captures or frames).

FIG. 19 illustrates an exemplary implementation 1902 of metadata thatmay be provided by virtual reality content provider system 1604 andreceived by media player device 1602. Implementation 1902 may adhere toa JSON data format, a binary data format, or any other suitable dataformat as may serve a particular implementation. FIG. 19 shows exemplarydata fields that may be used to define metadata associated with aparticular capture device (e.g., capture device 202-1). A briefdescription of each of the fields shown in FIG. 19 will now be provided.

The field labeled “stream IDs” in the example shown in FIG. 19 may bepopulated with an identifier for the color video data stream and anidentifier for the depth video data stream associated with a particularcapture device. The identifiers may be represented using a string typevariable or any other suitable data type. The identifiers may be in anysuitable format for any suitable type of data stream. For example, foran MPEG transport stream, the identifiers may be packet identifiers(“PIDs”) for packets in the MPEG transport stream.

The field labeled “depth mapping” in the example shown in FIG. 19 may bepopulated with “near” and “far” depth values provided initially bycapture facility 106 and representative of a distance (e.g., from theparticular capture device) of a minimum value of a representable rangeof values and a distance (e.g., from the particular capture device) of amaximum value of the representable range of values, respectively. The“near” and “far” depth values may map data values to real-world unitsand may be represented by floating point values or any other suitabledata type and may represent a depth mapping range, such as a depthmapping range bounded by a near plane and a far plane located at thespecified depth values. In an embodiment, this may include a moregeneralized “depth decode” in a format such as

$\begin{bmatrix}{zw} \\w\end{bmatrix} = {\begin{bmatrix}a & b \\c & d\end{bmatrix}\begin{bmatrix}z^{\prime} \\1\end{bmatrix}}$

where z=(az′+b)/(cz′+d), and a capture system (e.g., one or more capturedevices such as capture devices 202 in FIG. 4) sends the matrixcoefficients a, b, c, d.

The field labeled “3×4 column-major transform matrix” in the exampleshown in FIG. 19 may be populated with a transformation matrix to beused by system 100 to transform data included in the color video datastream and the depth video data stream into 3D coordinates in a commonvirtual 3D space.

The field labeled “FOV tangent angles” in the example shown in FIG. 19may be populated with information indicating the field of view orintrinsic capture device parameters, such as those based on focallength, center point, etc., of the particular capture device in terms ofview angles or tangents of angles (e.g., tan_(left), tan_(right),tan_(top), and tan_(bottom)). In certain examples, metadata mayadditionally or alternatively include extrinsic capture deviceparameters and/or information related to linear and/or non-linear depthdisparity associated with capture devices and/or captured surface data.

Implementation 1102 shown in FIG. 11 is illustrative of one way thatmetadata may be represented and provided to media player device 210 incertain examples. In other examples, metadata may be represented in anyother format and may include any additional or alternative informationas may be suitable for a particular implementation.

Returning to FIG. 16, media player device 1602 may include or beimplemented by any device capable of receiving data representative of 2Dcolor data and depth data and metadata (e.g., in a transport stream) andprocessing the received data to generate virtual reality content asdescribed herein. Media player device 1602 may generate and use partial3D meshes such as those described herein to render and present an imageview of a virtual 3D space from a selected arbitrary viewpoint withinthe virtual 3D space, which image view may be a perspective field ofview of a virtual 3D space (e.g., a perspective view of a virtualreality world representative of a captured 3D scene). As shown in FIG.16, media player device 1602 may be operated by a user 1608. Mediaplayer device 1602 may detect user input from user 1608 and, based onthe user input, dynamically update the image view of the virtual 3Dspace to be rendered and presented as user 1608 experiences the virtualreality world represented in the virtual 3D space.

For example, a field of view from an arbitrary viewpoint within thevirtual reality world may provide an image view through which user 1608may easily and naturally look around the virtual reality world. Thefield of view may be presented by media player device 1602 (e.g., on adisplay screen of media player device 1602) and may include videodepicting objects surrounding the viewpoint of user 1608 within thevirtual reality world. Additionally, the field of view may dynamicallychange in response to user input provided by user 1608 as user 1608experiences the virtual reality world. For example, media player device1602 may detect user input (e.g., moving or turning the display screenupon which the field of view is presented). In response, the field ofview may display different objects and/or objects seen from a differentviewpoint (e.g., a viewpoint corresponding to the position of thedisplay screen) in place of the objects seen from the previousviewpoint.

To facilitate user 1608 in experiencing an image view of a virtual 3Dspace, media player device 1602 may include or be associated with atleast one display screen (e.g., a head-mounted display screen built intoa head-mounted virtual reality device or a display screen of a mobiledevice mounted to the head of the user with an apparatus such as acardboard apparatus) upon which image views of a virtual reality worldmay be displayed. Media player device 1602 may also include softwareconfigured to receive, maintain, and/or process 2D color data and depthdata representative of the virtual reality world, together withcorresponding metadata, to render and present views of the virtualreality world on the display screen(s) of media player device 1602 inaccordance with any of the operations of system 100 described herein.For example, media player device 1602 may include dedicated, standalonesoftware applications (e.g., mobile applications) configured to processand present data representative of a virtual reality world on thedisplay(s). In other examples, the software used to present theparticular views of the virtual reality world may include non-dedicatedsoftware such as a standard web browser application.

Additionally or alternatively, media player device 1602 may includehardware, firmware, and/or software configured to facilitate receiving,decoding, demultiplexing, and/or processing 2D color data, depth data,and metadata. For example, media player device 1602 may include adecoding/demultiplexing system 1610 that may perform any suitabledecoding and/or demultiplexing operations associated with 2D color data,depth data, and metadata received from virtual reality media providersystem 1604. In certain examples, decoding/demultiplexing system 1610may include a graphics card having dedicated video decoding hardware(e.g., one or more dedicated video decoders) and a programmable GPU. Thegraphics card may include a limited number of decoders capable ofdecoding received data. Accordingly, in certain examples, data that isreceived by media player device 1602 (e.g., data included in a transportstream that is received by media player device 1602) may be tailored byvirtual reality media provider system 1604 to certain hardwareconstraints of media player device 1602 (e.g., by limiting the number ofstreams included in the transport stream).

Decoding/demultiplexing system 1610 may be communicatively coupled tosystem 100 in any suitable manner. Accordingly, in certain examples,system 100 may receive demultiplexed and/or decoded 2D captured imagerydata and projection data from decoding/demultiplexing system 1610 andprocess the received data as described herein to render virtual realitycontent. It will be recognized that although systems 100 and 1610 areshown to be separate systems in FIG. 16, systems 100 and 1610 may becombined into fewer systems, such as into a single system, or dividedinto more systems as may serve a particular implementation.

Media player device 1602 may take one of several different form factors.For example, media player device 1602 may include or be implemented by ahead-mounted virtual reality device (e.g., a virtual reality gamingdevice) that includes a head-mounted display screen, by a personalcomputer device (e.g., a desktop computer, laptop computer, etc.), by amobile or wireless device (e.g., a smartphone, a tablet device, a mobilereader, etc.), or by any other device or configuration of devices thatmay serve a particular implementation to facilitate receiving 2D colordata, depth data, and metadata and rendering and presenting virtualreality content. Different types of media player devices (e.g.,head-mounted virtual reality devices, personal computer devices, mobiledevices, etc.) may provide different types of virtual realityexperiences having different levels of immersion for user 1608.

While examples of certain media player devices have been described, theexamples are illustrative and not limiting. A media player device mayinclude any suitable device and/or configuration of devices configuredto receive 2D color data, depth data, and metadata and use the 2D colordata, depth data, and metadata to render an image view of a virtual 3Dspace according to principles described herein. For example, a mediaplayer device may include a tethered device configuration (e.g., atethered headset device) or an untethered device configuration (e.g., adisplay screen untethered from a processing device). As another example,a head-mounted virtual reality media player device or other media playerdevice may be used in conjunction with a virtual reality controller suchas a wearable controller (e.g., a ring controller) and/or a handheldcontroller.

Although FIG. 16 shows system 100 as being part of media player device1602, it is understood that one or more facilities of system 100 may beincorporated in or distributed across other computing devices inalternative implementations. For example, one or more of the operationsperformed by rendering facility 102 may be performed by a computingdevice (e.g., server) associated with virtual reality media providersystem 1604 and/or another system that is separate from media playerdevice 1602. In addition, storage facility 104 may be located remotely(e.g., as a cloud-based storage facility) from media player device 1602in certain implementations.

FIG. 20 illustrates an exemplary method 2000 for rendering virtualreality content based on 2D captured imagery of a 3D scene. While FIG.20 illustrates exemplary operations according to one embodiment, otherembodiments may omit, add to, reorder, and/or modify any of theoperations shown in FIG. 20. One or more of the operations shown in FIG.20 may be performed by rendering system 100 and/or any implementationthereof.

In operation 2002, a virtual reality content rendering system (e.g.,virtual reality content rendering system 100) may receive 2D color dataand depth data captured by a plurality of capture devices disposed atdifferent vantage points in relation to a 3D scene. Operation 2002 maybe performed in any of the ways described herein.

In operation 2004, the virtual reality content rendering system mayreceive metadata for the 2D color data and the depth data. Operation2004 may be performed in any of the ways described herein.

In operation 2006, the virtual reality content rendering system maygenerate, for each vantage point associated with each respective capturedevice included in the plurality of capture devices, and based on themetadata and the depth data, a partial 3D mesh projected into a virtual3D space to produce a partial virtual representation of the 3D scene inthe virtual 3D space. Operation 2006 may be performed in any of the waysdescribed herein.

In operation 2008, the virtual reality content rendering system maygenerate, based on the partial 3D meshes projected into the virtual 3Dspace, and from an arbitrary viewpoint within the virtual 3D space, animage view of the virtual 3D space. In certain examples, the generatingof the image view may include accumulating the partial 3D meshesprojected into the virtual 3D space, and blending, based on the 2D colordata, color samples for the partial 3D meshes to form the image view ofthe virtual 3D space. Operation 2008 may be performed in any of the waysdescribed herein.

In certain examples, one or more operations of system 100 describedherein may be performed in accordance with, as part of, and/or using agraphics pipeline of a GPU and/or graphics card (e.g., an OpenGLrendering pipeline). Such implementations of system 100 may leverage oneor more capabilities, functions, and/or resources of the GPU and/orgraphics card (e.g., existing video codecs designed to represent 2Ddata), including fixed functions, hardware-accelerated functions, and/orbuffers of the GPU and/or graphics card (e.g., fixed-functionrasterization, hardware-accelerated depth testing, a depth buffer,etc.).

In certain examples, a graphics pipeline may include at least vertexprocessing, rasterization, and fragment processing stages, and one ormore operations of system 100 may be performed within these stages ofthe graphics pipeline. As an example, system 100 may generate partial 3Dmeshes, as described herein, within a vertex processing stage of thegraphics pipeline. As another example, system 100 may project 2D inputmeshes to generate partial 3D meshes in a virtual 3D space, as describedherein, using a vertex shader within the vertex processing stage of thegraphics pipeline. As another example, system 100 may remove extraneousprimitives of the partial 3D meshes and/or determine blend weights forprimitives of the partial 3D meshes, as described herein,pre-rasterization using a geometry shader and/or a tessellation shaderwithin the vertex processing stage of the graphics pipeline and/orpost-rasterization using a fragment shader within a fragment processingstage of the graphics pipeline. As another example, system 100 maysample and blend color samples, as described herein, post-rasterizationusing a fragment shader within a fragment processing stage of thegraphics pipeline.

In certain examples, system 100 may perform multiple passes on a dataset (e.g., multiple passes on 2D color data, depth data, and metadatafor one frame in a sequence of capture frames) in a way that leverages agraphics pipeline. For example, system 100 may iterate on the data setonce to perform a depth pre-pass to establish an accumulation regionwithin which to accumulate color samples, and may iterate on the dataset again to accumulate color samples within the accumulation region.System 100 may iterate on the accumulated color samples to blend thecolor samples to form an image view that may be output for display.

FIG. 21 illustrates an exemplary method for rendering virtual realitycontent based on 2D captured imagery of a 3D scene according toprinciples described herein. While FIG. 21 illustrates exemplaryoperations according to one embodiment, other embodiments may omit, addto, reorder, and/or modify any of the operations shown in FIG. 21. Oneor more of the operations shown in FIG. 21 may be performed by a virtualreality content rendering system (e.g., virtual reality contentrendering system 100) and/or by any implementation or combinationthereof.

In operation 2102, a virtual reality content rendering system (e.g.,virtual reality content rendering system 100) may receive 2D color dataand depth data captured by a plurality of capture devices disposed atdifferent vantage points in relation to a 3D scene. Operation 2102 maybe performed in any of the ways described herein.

In operation 2104, the virtual reality content rendering system mayreceive metadata for the 2D color data and the depth data. Operation2104 may be performed in any of the ways described herein.

In operation 2106, the virtual reality content rendering system mayperform a depth pass (e.g., a depth pre-pass). The depth pass mayinclude the virtual reality content rendering system transformingreceived depth data (e.g., captured depth data representing positions ofobjects in the 3D scene) and metadata (e.g., vantage point information)into 3D coordinates in a common virtual 3D space and projecting 2D inputmeshes for the vantage points into 3D coordinates in the virtual 3Dspace to form partial 3D meshes projected into the virtual 3D space, asdescribed herein. The depth pass may also include the virtual realitycontent rendering system culling depth data, such as by depth testing,from the perspective of a selected arbitrary viewpoint within thevirtual 3D space, to identify and omit occluded samples.

The depth pass may also include the virtual reality content renderingsystem determining, from a selected arbitrary viewpoint within virtual3D space, an accumulation range within which color samples will beaccumulated. The virtual reality content rendering system may determinethe accumulation range by offsetting depth values nearest to theselected arbitrary viewpoint by translating along perspective from theselected arbitrary viewpoint, such as described herein. The virtualreality content rendering system may store data representative of theaccumulation range, such as data representing a far value of theaccumulation range, for use in the accumulation pass.

The depth pass may be set up for a depth test and a depth write, with acolor write disabled. Accordingly, the virtual reality content renderingsystem may write data to a depth buffer during the depth pass. The datawritten to the depth buffer may include front-most depth sample valuesoffset by the predetermined depth offset (i.e., the far value of theaccumulation range). The virtual reality content rendering system may beconfigured not to clear the depth buffer before the accumulation passsuch that the depth buffer may provide input to fixed-function depthtesting performed in the accumulation pass. This may providehardware-accelerated rejection of samples outside of the determinedaccumulation range (e.g., samples beyond the far value of theaccumulation range), and samples that are outside of the accumulationrange (e.g., samples beyond the depth offset) will not be output fromrasterization. Samples that are within the accumulation range (i.e.,samples in front of the far value of the accumulation range) will beselected and contribute to the drawn image view.

In operation 2108, the virtual reality content rendering system mayperform an accumulation pass. The accumulation pass may include thevirtual reality content rendering system transforming received depthdata (e.g., captured depth data representing positions of objects in the3D scene) and metadata (e.g., vantage point field of view coordinates)into 3D coordinates in a common virtual 3D space and projecting 2D inputmeshes for the vantage points into 3D coordinates in the virtual 3Dspace to form partial 3D meshes projected into the virtual 3D space, asdescribed herein. The accumulation pass may also include the virtualreality content rendering system culling depth data, such as by depthtesting, from the perspective of the selected arbitrary viewpoint withinthe virtual 3D space, to identify and omit occluded samples. Theaccumulation pass may also include the virtual reality content renderingsystem determining blend weights for primitives of the partial 3Dmeshes, such as described herein.

The accumulation pass may also include the virtual reality contentrendering system rasterizing a perspective image view of the projectedpartial 3D meshes from the selected arbitrary viewpoint within thevirtual 3D space and accumulating color samples, from the 2D color data(e.g., from a color video stream at input 2D coordinates), for 3Dcoordinates of the projected partial 3D meshes that are within theaccumulation region established in the depth pass, as described herein.The color samples for the perspective image view may be accumulated in aframe buffer as described herein. In certain examples, the accumulatingmay use additive blending to accumulate a total of all fragment outputs,which blending may be a fixed-function operation in certainimplementations. Blend weights determined in the accumulation pass maybe used to multiply accumulated color samples that are then addedtogether to determine a weighted total for all fragments, such asdescribed herein.

The accumulation pass may be set up for a depth test and a color write,with a depth write disabled. Accordingly, the virtual reality contentrendering system may write accumulated color sample data to anintermediary, high-precision frame buffer during the accumulation pass,as described herein.

In operation 2110, the virtual reality content rendering system mayperform a weighted average division pass. The weighted average divisionpass may include the virtual reality content rendering system dividingaccumulated, weighted color samples by a total of the blend weights todetermine a weighted average of the accumulated color samples. Thevirtual reality content rendering system may draw a full-screen quad(e.g., a quad covering x/y [−1, 1] in clip coordinates) of thedetermined weighted averages to an output (e.g., display) frame bufferin step 2112. The content in the output frame buffer may be used by thevirtual reality content rendering system and/or a media player deviceimplementing the virtual reality content rendering system to display aperspective image view of the virtual 3D space that has been formed bythe virtual reality content rendering system generating, accumulating,and blending partial 3D meshes as described herein.

In certain examples, method 2000 or method 2100 may be performed for aframe in a temporal sequence of frames. The frame may include a 2D colordata frame, a depth data frame, and metadata associated with the frame(e.g., associated with a point in time corresponding to the frame).Accordingly, method 2000 or method 2100 may be repeated for each otherframe in the temporal sequence of frames to produce video output of adynamic perspective image view of a virtual 3D space representative of acaptured 3D scene.

In certain embodiments, one or more of the systems, components, and/orprocesses described herein may be implemented and/or performed by one ormore appropriately configured computing devices. To this end, one ormore of the systems and/or components described above may include or beimplemented by any computer hardware and/or computer-implementedinstructions (e.g., software) embodied on at least one non-transitorycomputer-readable medium configured to perform one or more of theprocesses described herein. In particular, system components may beimplemented on one physical computing device or may be implemented onmore than one physical computing device. Accordingly, system componentsmay include any number of computing devices, and may employ any of anumber of computer operating systems.

In certain embodiments, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices. In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions may be stored and/or transmittedusing any of a variety of known computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory medium that participates inproviding data (e.g., instructions) that may be read by a computer(e.g., by a processor of a computer). Such a medium may take many forms,including, but not limited to, non-volatile media, and/or volatilemedia. Non-volatile media may include, for example, optical or magneticdisks and other persistent memory. Volatile media may include, forexample, dynamic random access memory (“DRAM”), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a disk, hard disk, magnetic tape, any othermagnetic medium, a compact disc read-only memory (“CD-ROM”), a digitalvideo disc (“DVD”), any other optical medium, random access memory(“RAM”), programmable read-only memory (“PROM”), electrically erasableprogrammable read-only memory (“EPROM”), FLASH-EEPROM, any other memorychip or cartridge, or any other tangible medium from which a computercan read.

FIG. 22 illustrates an exemplary computing device 2200 that may bespecifically configured to perform one or more of the processesdescribed herein. As shown in FIG. 22, computing device 2200 may includea communication interface 2202, a processor 2204, a storage device 2206,an input/output (“I/O”) module 2208, and a graphics card 2210communicatively connected via a communication infrastructure 2212. Whilean exemplary computing device 2200 is shown in FIG. 22, the componentsillustrated in FIG. 22 are not intended to be limiting. Additional oralternative components may be used in other embodiments. Components ofcomputing device 2200 shown in FIG. 22 will now be described inadditional detail.

Communication interface 2202 may be configured to communicate with oneor more computing devices. Examples of communication interface 2202include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 2204 generally represents any type or form of processing unit(e.g., a central processing unit) capable of processing data orinterpreting, executing, and/or directing execution of one or more ofthe instructions, processes, and/or operations described herein.Processor 2204 may direct execution of operations in accordance with oneor more applications 2214 or other computer-executable instructions suchas may be stored in storage device 2206 or another computer-readablemedium.

Storage device 2206 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 2206 mayinclude, but is not limited to, a hard drive, network drive, flashdrive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatileand/or volatile data storage units, or a combination or sub-combinationthereof. Electronic data, including data described herein, may betemporarily and/or permanently stored in storage device 2206. Forexample, data representative of one or more executable applications 2214configured to direct processor 2204 to perform any of the operationsdescribed herein may be stored within storage device 2206. In someexamples, data may be arranged in one or more databases residing withinstorage device 2206. In certain examples, storage device 2206 maymaintain surface data, metadata, data streams, video streams, transportstreams, and/or any other data received, generated, managed, maintained,used, and/or transmitted to or by facility 102. Storage device 2206 mayfurther include any other data as may be used by facility 102 to performone of more of the operations described herein.

I/O module 2208 may include one or more I/O modules configured toreceive user input and provide user output. One or more I/O modules maybe used to receive input for a single virtual reality experience. I/Omodule 2208 may include any hardware, firmware, software, or combinationthereof supportive of input and output capabilities. For example, I/Omodule 2208 may include hardware and/or software for capturing userinput, including, but not limited to, a keyboard or keypad, atouchscreen component (e.g., touchscreen display), a receiver (e.g., anRF or infrared receiver), motion sensors, and/or one or more inputbuttons.

I/O module 2208 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 2208 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation. I/O module2208 may be omitted from certain implementations.

In some examples, any of the facilities described herein may beimplemented by or within one or more components of computing device2200. For example, one or more applications 2214 residing within storagedevice 2206 may be configured to direct processor 2204 to perform one ormore processes or functions associated with rendering facility 102 ofvirtual reality content rendering system 100 (see FIG. 1).Alternatively, one or more applications 2214 residing within storagedevice 2206 may be configured to direct processor 2204 to perform one ormore processes or functions associated with data decoding/demultiplexingsystem 1610 (see FIG. 16). Likewise, storage facility 104 of system 100may be implemented by or within storage device 2206.

Graphics card 2210 may include any suitable graphics card (e.g., acommercially available graphics card) having dedicated video decodinghardware (e.g., one or more dedicated video decoders) and a programmableGPU 2216. Graphics card 2210 may include additional components incertain embodiments. Graphics card 2210 and/or GPU 2216 may beconfigured to execute and/or assist processor 2204 in executing one ormore of the exemplary operations described herein. Graphics card 2210may include any suitable number of graphics cards and/or GPUs as maysuit a particular implementation.

To the extent the aforementioned embodiments collect, store, and/oremploy personal information provided by individuals, it should beunderstood that such information shall be used in accordance with allapplicable laws concerning protection of personal information.Additionally, the collection, storage, and use of such information maybe subject to consent of the individual to such activity, for example,through well known “opt-in” or “opt-out” processes as may be appropriatefor the situation and type of information. Storage and use of personalinformation may be in an appropriately secure manner reflective of thetype of information, for example, through various encryption andanonymization techniques for particularly sensitive information.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: receiving, by a virtualreality content rendering system, two-dimensional (“2D”) color data anddepth data captured by a plurality of capture devices disposed atdifferent vantage points in relation to a three-dimensional (“3D”)scene; receiving, by the virtual reality content rendering system,metadata for the 2D color data and the depth data; generating, by thevirtual reality content rendering system, for each vantage pointassociated with each respective capture device included in the pluralityof capture devices, and based on the metadata and the depth data, apartial 3D mesh projected into a virtual 3D space to produce a partialrepresentation of the 3D scene in the virtual 3D space; and generating,by the virtual reality content rendering system based on the partial 3Dmeshes projected into the virtual 3D space, and from an arbitraryviewpoint within the virtual 3D space, an image view of the virtual 3Dspace, the generating of the image view comprising accumulating thepartial 3D meshes projected into the virtual 3D space, and blending,based on the 2D color data, color samples for overlapping sections ofthe partial 3D meshes to form the image view of the virtual 3D space. 2.The method of claim 1, wherein the generating of the partial 3D mesh foreach vantage point associated with each respective capture deviceincluded in the plurality of capture devices comprises: projecting, foreach vantage point associated with each respective capture deviceincluded in the plurality of capture devices, a 2D input mesh alongperspective rays to depths within the virtual 3D space based on thedepth data to produce 3D coordinates of vertices of primitives of thepartial 3D mesh in the virtual 3D space.
 3. The method of claim 2,wherein the generating of the partial 3D mesh for each vantage pointassociated with each respective capture device included in the pluralityof capture devices further comprises: identifying, for each partial 3Dmesh, one or more of the primitives of the partial 3D mesh as extraneousprimitives; and removing the identified extraneous primitives from eachpartial 3D mesh.
 4. The method of claim 3, wherein the identifying, foreach partial 3D mesh, one or more of the primitives of the partial 3Dmesh as extraneous primitives comprises at least one of: identifying afirst primitive of the partial 3D mesh as a first extraneous primitivebased at least in part on the first primitive having a depth change thatexceeds a predetermined depth change threshold; and identifying a secondprimitive of the partial 3D mesh as a second extraneous primitive basedat least in part on the second primitive having a surface not facing therespective vantage point associated with the partial 3D mesh.
 5. Themethod of claim 2, wherein the blending of the color samples for theoverlapping sections of the partial 3D meshes to form the image view ofthe virtual 3D space comprises: determining blend weights for theprimitives of the partial 3D meshes; weighting the color samples for thepartial 3D meshes based on the determined blend weights for theprimitives of the partial 3D meshes; and determining weighted averagesof the weighted color samples for superimposed primitives included inthe overlapping sections of the partial 3D meshes.
 6. The method ofclaim 1, wherein: the accumulating of the partial 3D meshes projectedinto the virtual 3D space comprises accumulating the color samples forthe partial 3D meshes in a frame buffer; and the blending of the colorsamples for the overlapping sections of the partial 3D meshes to formthe image view of the virtual 3D space comprises additively blending thecolor samples for the overlapping sections of the partial 3D meshes inthe frame buffer.
 7. The method of claim 1, further comprising:determining, by the virtual reality content rendering system based onthe partial 3D meshes projected into the virtual 3D space, and from thearbitrary viewpoint within the virtual 3D space, an accumulation region;wherein the accumulating of the partial 3D meshes and the blending ofthe color samples for the overlapping sections of the partial 3D meshesto form the image view of the virtual 3D space are performed only forfragments that are within the accumulation region.
 8. The method ofclaim 1, embodied as computer-executable instructions on at least onenon-transitory computer-readable medium.
 9. A non-transitorycomputer-readable medium storing instructions that, when executed,direct at least one processor of a computing device to: receivetwo-dimensional (“2D”) color data and depth data captured by a pluralityof capture devices disposed at different vantage points in relation to athree-dimensional (“3D”) scene, receive metadata for the 2D color dataand the depth data, generate, for each vantage point associated witheach respective capture device included in the plurality of capturedevices, and based on the metadata and the depth data, a partial 3D meshprojected into a virtual 3D space to produce a partial representation ofthe 3D scene in the virtual 3D space, and generate, based on the partial3D meshes projected into the virtual 3D space, and from an arbitraryviewpoint within the virtual 3D space, an image view of the virtual 3Dspace, the generating of the image view comprising blending, based onthe 2D color data, color samples for overlapping sections of the partial3D meshes to form the image view of the virtual 3D space.
 10. Thenon-transitory computer-readable medium of claim 9, wherein thegenerating of the partial 3D mesh for each vantage point associated witheach respective capture device included in the plurality of capturedevices comprises: the instructions, when executed, directing the atleast one processor to project, for each vantage point associated witheach respective capture device included in the plurality of capturedevices, a 2D input mesh along perspective rays to depths within thevirtual 3D space based on the depth data to produce 3D coordinates ofvertices of primitives of the partial 3D mesh in the virtual 3D space.11. The non-transitory computer-readable medium of claim 10, wherein thegenerating of the partial 3D mesh for each vantage point associated witheach respective capture device included in the plurality of capturedevices further comprises the instructions, when executed, directing theat least one processor to: identify, for each partial 3D mesh, one ormore of the primitives of the partial 3D mesh as extraneous primitives;and remove the identified extraneous primitives from each partial 3Dmesh.
 12. The non-transitory computer-readable medium of claim 10,wherein the blending of the color samples for the overlapping sectionsof the partial 3D meshes to form the image view of the virtual 3D spacecomprises the instructions, when executed, directing the at least oneprocessor to: determine blend weights for the primitives of the partial3D meshes; weight the color samples for the partial 3D meshes based onthe determined blend weights for the primitives of the partial 3Dmeshes; and determine weighted averages of the weighted color samplesfor superimposed primitives included in the overlapping sections of thepartial 3D meshes.
 13. The non-transitory computer-readable medium ofclaim 9, wherein the blending of the color samples for the overlappingsections of the partial 3D meshes to form the image view of the virtual3D space comprises the instructions, when executed, directing the atleast one processor to accumulate the color samples for the partial 3Dmeshes in a frame buffer.
 14. The non-transitory computer-readablemedium of claim 9, further comprising instructions that, when executed,direct the at least one processor to determine, based on the partial 3Dmeshes projected into the virtual 3D space, and from the arbitraryviewpoint within the virtual 3D space, an accumulation region; whereinthe blending of the color samples for the overlapping sections of thepartial 3D meshes to form the image view of the virtual 3D space isperformed only for color samples that are within the accumulationregion.
 15. A system comprising: at least one computer processor; and avirtual reality rendering facility that directs the at least onecomputer processor to: receive two-dimensional (“2D”) color data anddepth data captured by a plurality of capture devices disposed atdifferent vantage points in relation to a three-dimensional (“3D”)scene, receive metadata for the 2D color data and the depth data,generate, for each vantage point associated with each respective capturedevice included in the plurality of capture devices, and based on themetadata and the depth data, a partial 3D mesh projected into a virtual3D space to produce a partial representation of the 3D scene in thevirtual 3D space, and generate, based on the partial 3D meshes projectedinto the virtual 3D space, and from an arbitrary viewpoint within thevirtual 3D space, an image view of the virtual 3D space, the generatingof the image view comprising blending, based on the 2D color data, colorsamples for overlapping sections of the partial 3D meshes to form theimage view of the virtual 3D space.
 16. The system of claim 15, whereinthe generating of the partial 3D mesh for each vantage point associatedwith each respective capture device included in the plurality of capturedevices comprises: the virtual reality rendering facility directing theat least one computer processor to project, for each vantage pointassociated with each respective capture device included in the pluralityof capture devices, a 2D input mesh along perspective rays to depthswithin the virtual 3D space based on the depth data to produce 3Dcoordinates of vertices of primitives of the partial 3D mesh in thevirtual 3D space.
 17. The system of claim 16, wherein the generating ofthe partial 3D mesh for each vantage point associated with eachrespective capture device included in the plurality of capture devicesfurther comprises: the virtual reality rendering facility directing theat least one computer processor to identify, for each partial 3D mesh,one or more of the primitives of the partial 3D mesh as extraneousprimitives; and the virtual reality rendering facility directing the atleast one computer processor to remove the identified extraneousprimitives from each partial 3D mesh.
 18. The system of claim 16,wherein the blending of the color samples for the overlapping sectionsof the partial 3D meshes to form the image view of the virtual 3D spacecomprises: the virtual reality rendering facility directing the at leastone computer processor to determine blend weights for the primitives ofthe partial 3D meshes; the virtual reality rendering facility directingthe at least one computer processor to weight the color samples for thepartial 3D meshes based on the determined blend weights for theprimitives of the partial 3D meshes; and the virtual reality renderingfacility directing the at least one computer processor to determineweighted averages of the weighted color samples for superimposedprimitives included in overlapping areas of the partial 3D meshes. 19.The system of claim 15, wherein the blending of the color samples forthe overlapping sections of the partial 3D meshes to form the image viewof the virtual 3D space comprises the virtual reality rendering facilitydirecting the at least one computer processor to accumulate the colorsamples for the partial 3D meshes in a frame buffer.
 20. The system ofclaim 15, further comprising the virtual reality rendering facilitydirecting the at least one computer processor to determine, based on thepartial 3D meshes projected into the virtual 3D space, and from thearbitrary viewpoint within the virtual 3D space, an accumulation region;wherein the blending of the color samples for the overlapping sectionsof the partial 3D meshes to form the image view of the virtual 3D spaceis performed only for color samples that are within the accumulationregion.